kotovalexarian-likes-gitlab
/
gitlab-org--gitlab-foss
1
0
Fork 0
Code Releases Activity
gitlab-org--gitlab-foss/lib/backup/files.rb

77 lines
2.6 KiB
Ruby
Raw Normal View History

Use tar for intermediate backup storage During the backup we create an intermediate copy of two directories: builds and uploads. Instead of creating many small files with 'cp -r', we now use tar (and fast gzip) to create single intermediate files. This saves on disk IO and disk space while creating a backup.
2015-10-06 12:43:19 +00:00
require 'open3'
Move existing dirs to a temp location while restore Instead of renaming existing storage directories to a timestamped format, create a timestamped directory in a temporary location and move the individual files and folders inside storage directories to it. This helps us with nested storage directories.
2018-03-05 08:25:02 +00:00
require_relative 'helper'
Use tar for intermediate backup storage During the backup we create an intermediate copy of two directories: builds and uploads. Instead of creating many small files with 'cp -r', we now use tar (and fast gzip) to create single intermediate files. This saves on disk IO and disk space while creating a backup.
2015-10-06 12:43:19 +00:00
module Backup
class Files
Move existing dirs to a temp location while restore Instead of renaming existing storage directories to a timestamped format, create a timestamped directory in a temporary location and move the individual files and folders inside storage directories to it. This helps us with nested storage directories.
2018-03-05 08:25:02 +00:00
include Backup::Helper
Remove unused variable
2015-10-06 13:22:08 +00:00
attr_reader :name, :app_files_dir, :backup_tarball, :files_parent_dir
Use tar for intermediate backup storage During the backup we create an intermediate copy of two directories: builds and uploads. Instead of creating many small files with 'cp -r', we now use tar (and fast gzip) to create single intermediate files. This saves on disk IO and disk space while creating a backup.
2015-10-06 12:43:19 +00:00
Do no rely on basename of builds, uploads
2015-10-16 15:16:17 +00:00
def initialize(name, app_files_dir)
@name = name
Use tar for intermediate backup storage During the backup we create an intermediate copy of two directories: builds and uploads. Instead of creating many small files with 'cp -r', we now use tar (and fast gzip) to create single intermediate files. This saves on disk IO and disk space while creating a backup.
2015-10-06 12:43:19 +00:00
@app_files_dir = File.realpath(app_files_dir)
@files_parent_dir = File.realpath(File.join(@app_files_dir, '..'))
# This is a combination of 2 commits. # This is the 1st commit message: Add `copy` backup strategy to combat file changed errors The backup Rake task used to stream data directly from the live data directory into the backup. Under many circumstances this worked OK. However, really active instances would experience a 'file changed as we read it' error - especially with data like the registry. This now copies the data first, then compresses it. It will take a bit more disk space while the backup is in progress, but it's a necessary thing. # The commit message #2 will be skipped: # Add env var
2017-01-23 22:42:34 +00:00
@backup_files_dir = File.join(Gitlab.config.backup.path, File.basename(@app_files_dir) )
Remove unused variable
2015-10-06 13:22:08 +00:00
@backup_tarball = File.join(Gitlab.config.backup.path, name + '.tar.gz')
Use tar for intermediate backup storage During the backup we create an intermediate copy of two directories: builds and uploads. Instead of creating many small files with 'cp -r', we now use tar (and fast gzip) to create single intermediate files. This saves on disk IO and disk space while creating a backup.
2015-10-06 12:43:19 +00:00
end
# Copy files from public/files to backup/files
def dump
FileUtils.mkdir_p(Gitlab.config.backup.path)
Remove old "files" tarball explicitly
2015-10-06 13:10:13 +00:00
FileUtils.rm_f(backup_tarball)
# This is a combination of 2 commits. # This is the 1st commit message: Add `copy` backup strategy to combat file changed errors The backup Rake task used to stream data directly from the live data directory into the backup. Under many circumstances this worked OK. However, really active instances would experience a 'file changed as we read it' error - especially with data like the registry. This now copies the data first, then compresses it. It will take a bit more disk space while the backup is in progress, but it's a necessary thing. # The commit message #2 will be skipped: # Add env var
2017-01-23 22:42:34 +00:00
if ENV['STRATEGY'] == 'copy'
Ignore "lost+found" folder during backup on a volume
2017-12-20 08:11:13 +00:00
cmd = %W(rsync -a --exclude=lost+found #{app_files_dir} #{Gitlab.config.backup.path})
# This is a combination of 2 commits. # This is the 1st commit message: Add `copy` backup strategy to combat file changed errors The backup Rake task used to stream data directly from the live data directory into the backup. Under many circumstances this worked OK. However, really active instances would experience a 'file changed as we read it' error - especially with data like the registry. This now copies the data first, then compresses it. It will take a bit more disk space while the backup is in progress, but it's a necessary thing. # The commit message #2 will be skipped: # Add env var
2017-01-23 22:42:34 +00:00
output, status = Gitlab::Popen.popen(cmd)
unless status.zero?
puts output
Find and mark more Git disk access locations
2018-06-05 15:51:14 +00:00
raise Backup::Error, 'Backup failed'
# This is a combination of 2 commits. # This is the 1st commit message: Add `copy` backup strategy to combat file changed errors The backup Rake task used to stream data directly from the live data directory into the backup. Under many circumstances this worked OK. However, really active instances would experience a 'file changed as we read it' error - especially with data like the registry. This now copies the data first, then compresses it. It will take a bit more disk space while the backup is in progress, but it's a necessary thing. # The commit message #2 will be skipped: # Add env var
2017-01-23 22:42:34 +00:00
end
Consistently use tar lookup function
2018-06-06 09:49:34 +00:00
run_pipeline!([%W(#{tar} --exclude=lost+found -C #{@backup_files_dir} -cf - .), %w(gzip -c -1)], out: [backup_tarball, 'w', 0600])
# This is a combination of 2 commits. # This is the 1st commit message: Add `copy` backup strategy to combat file changed errors The backup Rake task used to stream data directly from the live data directory into the backup. Under many circumstances this worked OK. However, really active instances would experience a 'file changed as we read it' error - especially with data like the registry. This now copies the data first, then compresses it. It will take a bit more disk space while the backup is in progress, but it's a necessary thing. # The commit message #2 will be skipped: # Add env var
2017-01-23 22:42:34 +00:00
FileUtils.rm_rf(@backup_files_dir)
else
Consistently use tar lookup function
2018-06-06 09:49:34 +00:00
run_pipeline!([%W(#{tar} --exclude=lost+found -C #{app_files_dir} -cf - .), %w(gzip -c -1)], out: [backup_tarball, 'w', 0600])
# This is a combination of 2 commits. # This is the 1st commit message: Add `copy` backup strategy to combat file changed errors The backup Rake task used to stream data directly from the live data directory into the backup. Under many circumstances this worked OK. However, really active instances would experience a 'file changed as we read it' error - especially with data like the registry. This now copies the data first, then compresses it. It will take a bit more disk space while the backup is in progress, but it's a necessary thing. # The commit message #2 will be skipped: # Add env var
2017-01-23 22:42:34 +00:00
end
Use tar for intermediate backup storage During the backup we create an intermediate copy of two directories: builds and uploads. Instead of creating many small files with 'cp -r', we now use tar (and fast gzip) to create single intermediate files. This saves on disk IO and disk space while creating a backup.
2015-10-06 12:43:19 +00:00
end
def restore
backup_existing_files_dir
Find and mark more Git disk access locations
2018-06-05 15:51:14 +00:00
run_pipeline!([%w(gzip -cd), %W(#{tar} --unlink-first --recursive-unlink -C #{app_files_dir} -xf -)], in: backup_tarball)
end
def tar
Consistently use tar lookup function
2018-06-06 09:49:34 +00:00
if system(*%w[gtar --version], out: '/dev/null')
# It looks like we can get GNU tar by running 'gtar'
'gtar'
else
'tar'
end
Use tar for intermediate backup storage During the backup we create an intermediate copy of two directories: builds and uploads. Instead of creating many small files with 'cp -r', we now use tar (and fast gzip) to create single intermediate files. This saves on disk IO and disk space while creating a backup.
2015-10-06 12:43:19 +00:00
end
def backup_existing_files_dir
Move existing dirs to a temp location while restore Instead of renaming existing storage directories to a timestamped format, create a timestamped directory in a temporary location and move the individual files and folders inside storage directories to it. This helps us with nested storage directories.
2018-03-05 08:25:02 +00:00
timestamped_files_path = File.join(Gitlab.config.backup.path, "tmp", "#{name}.#{Time.now.to_i}")
Use `File::exist?` instead of `File::exists?` Since version ruby-2.2.0, method `File::exists?` is deprecated.
2016-08-09 21:23:25 +00:00
if File.exist?(app_files_dir)
Move existing dirs to a temp location while restore Instead of renaming existing storage directories to a timestamped format, create a timestamped directory in a temporary location and move the individual files and folders inside storage directories to it. This helps us with nested storage directories.
2018-03-05 08:25:02 +00:00
# Move all files in the existing repos directory except . and .. to
# repositories.old.<timestamp> directory
FileUtils.mkdir_p(timestamped_files_path, mode: 0700)
files = Dir.glob(File.join(app_files_dir, "*"), File::FNM_DOTMATCH) - [File.join(app_files_dir, "."), File.join(app_files_dir, "..")]
begin
FileUtils.mv(files, timestamped_files_path)
rescue Errno::EACCES
access_denied_error(app_files_dir)
Describe workaround when restore fails because of `Errno::EBUSY` When `Errno::EBUSY` is raised during restore, this could indicate that the directory being restored into is a mountpoint. In this case we explain the user how to retry the restore.
2018-04-23 09:40:55 +00:00
rescue Errno::EBUSY
resource_busy_error(app_files_dir)
Move existing dirs to a temp location while restore Instead of renaming existing storage directories to a timestamped format, create a timestamped directory in a temporary location and move the individual files and folders inside storage directories to it. This helps us with nested storage directories.
2018-03-05 08:25:02 +00:00
end
Use tar for intermediate backup storage During the backup we create an intermediate copy of two directories: builds and uploads. Instead of creating many small files with 'cp -r', we now use tar (and fast gzip) to create single intermediate files. This saves on disk IO and disk space while creating a backup.
2015-10-06 12:43:19 +00:00
end
end
Don’t exclude some file in lib from rubocop
2017-02-22 18:18:40 +00:00
def run_pipeline!(cmd_list, options = {})
Use tar for intermediate backup storage During the backup we create an intermediate copy of two directories: builds and uploads. Instead of creating many small files with 'cp -r', we now use tar (and fast gzip) to create single intermediate files. This saves on disk IO and disk space while creating a backup.
2015-10-06 12:43:19 +00:00
status_list = Open3.pipeline(*cmd_list, options)
Find and mark more Git disk access locations
2018-06-05 15:51:14 +00:00
raise Backup::Error, 'Backup failed' unless status_list.compact.all?(&:success?)
Use tar for intermediate backup storage During the backup we create an intermediate copy of two directories: builds and uploads. Instead of creating many small files with 'cp -r', we now use tar (and fast gzip) to create single intermediate files. This saves on disk IO and disk space while creating a backup.
2015-10-06 12:43:19 +00:00
end
end
end