From 5516b6c47fa493cb792ffe501221fe397d1500ae Mon Sep 17 00:00:00 2001 From: Drew Blessing Date: Tue, 8 Mar 2016 16:52:48 -0600 Subject: [PATCH] Reload the schema before restoring a database backup --- CHANGELOG | 1 + doc/raketasks/backup_restore.md | 3 + doc/update/README.md | 1 + doc/update/restore_after_failure.md | 83 +++++++++++++++++++++++++++ lib/tasks/gitlab/backup.rake | 34 ++++++++--- lib/tasks/gitlab/db.rake | 35 +++++++++++ spec/tasks/gitlab/backup_rake_spec.rb | 40 +++++++------ 7 files changed, 171 insertions(+), 26 deletions(-) create mode 100644 doc/update/restore_after_failure.md create mode 100644 lib/tasks/gitlab/db.rake diff --git a/CHANGELOG b/CHANGELOG index 20a21abfb69..3e3385d769b 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -24,6 +24,7 @@ v 8.6.0 (unreleased) - Don't load all of GitLab in mail_room - HTTP error pages work independently from location and config (Artem Sidorenko) - Update `omniauth-saml` to 1.5.0 to allow for custom response attributes to be set + - Add option to reload the schema before restoring a database backup. !2807 - Memoize @group in Admin::GroupsController (Yatish Mehta) - Indicate how much an MR diverged from the target branch (Pierre de La Morinerie) - Added omniauth-auth0 Gem (Daniel Carraro) diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index f6d1234ac4a..4329ac30a1c 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -249,6 +249,9 @@ reconfigure` after changing `gitlab-secrets.json`. ### Installation from source ``` +# Stop processes that are connected to the database +sudo service gitlab stop + bundle exec rake gitlab:backup:restore RAILS_ENV=production ``` diff --git a/doc/update/README.md b/doc/update/README.md index 109d5de3fa2..0241f036830 100644 --- a/doc/update/README.md +++ b/doc/update/README.md @@ -15,3 +15,4 @@ Depending on the installation method and your GitLab version, there are multiple - [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostgreSQL. - [MySQL installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/database_mysql.md) contains additional information about configuring GitLab to work with a MySQL database. +- [Restoring from backup after a failed upgrade](restore_after_failure.md) diff --git a/doc/update/restore_after_failure.md b/doc/update/restore_after_failure.md new file mode 100644 index 00000000000..01c52aae7f5 --- /dev/null +++ b/doc/update/restore_after_failure.md @@ -0,0 +1,83 @@ +# Restoring from backup after a failed upgrade + +Upgrades are usually smooth and restoring from backup is a rare occurrence. +However, it's important to know how to recover when problems do arise. + +## Roll back to an earlier version and restore a backup + +In some cases after a failed upgrade, the fastest solution is to roll back to +the previous version you were using. + +First, roll back the code or package. For source installations this involves +checking out the older version (branch or tag). For Omnibus installations this +means installing the older .deb or .rpm package. Then, restore from a backup. +Follow the instructions in the +[Backup and Restore](../raketasks/backup_restore.md#restore-a-previously-created-backup) +documentation. + +## Potential problems on the next upgrade + +When a rollback is necessary it can produce problems on subsequent upgrade +attempts. This is because some tables may have been added during the failed +upgrade. If these tables are still present after you restore from the +older backup it can lead to migration failures on future upgrades. + +Starting in GitLab 8.6 we drop all tables prior to importing the backup to +prevent this problem. If you've restored a backup to a version prior to 8.6 you +may need to manually correct the problem next time you upgrade. + +Example error: + +``` +== 20151103134857 CreateLfsObjects: migrating ================================= +-- create_table(:lfs_objects) +rake aborted! +StandardError: An error has occurred, this and all later migrations canceled: + +PG::DuplicateTable: ERROR: relation "lfs_objects" already exists +``` + +Copy the version from the error. In this case the version number is +`20151103134857`. + +>**WARNING:** Use the following steps only if you are certain this is what you +need to do. + +### GitLab 8.6+ + +Pass the version to a database rake task to manually mark the migration as +complete. + +``` +# Source install +sudo -u git -H bundle exec rake gitlab:db:mark_migration_complete[20151103134857] RAILS_ENV=production + +# Omnibus install +sudo gitlab-rake gitlab:db:mark_migration_complete[20151103134857] +``` + +Once the migration is successfully marked, run the rake `db:migrate` task again. +You will likely have to repeat this process several times until all failed +migrations are marked complete. + +### GitLab < 8.6 + +``` +# Source install +sudo -u git -H bundle exec rails console production + +# Omnibus install +sudo gitlab-rails console +``` + +At the Rails console, type the following commands: + +``` +ActiveRecord::Base.connection.execute("INSERT INTO schema_migrations (version) VALUES('20151103134857')") +exit +``` + +Once the migration is successfully marked, run the rake `db:migrate` task again. +You will likely have to repeat this process several times until all failed +migrations are marked complete. + diff --git a/lib/tasks/gitlab/backup.rake b/lib/tasks/gitlab/backup.rake index cb4abe13799..402bb338f27 100644 --- a/lib/tasks/gitlab/backup.rake +++ b/lib/tasks/gitlab/backup.rake @@ -22,7 +22,7 @@ namespace :gitlab do end # Restore backup of GitLab system - desc "GitLab | Restore a previously created backup" + desc 'GitLab | Restore a previously created backup' task restore: :environment do warn_user_is_not_gitlab configure_cron_mode @@ -30,13 +30,31 @@ namespace :gitlab do backup = Backup::Manager.new backup.unpack - Rake::Task["gitlab:backup:db:restore"].invoke unless backup.skipped?("db") - Rake::Task["gitlab:backup:repo:restore"].invoke unless backup.skipped?("repositories") - Rake::Task["gitlab:backup:uploads:restore"].invoke unless backup.skipped?("uploads") - Rake::Task["gitlab:backup:builds:restore"].invoke unless backup.skipped?("builds") - Rake::Task["gitlab:backup:artifacts:restore"].invoke unless backup.skipped?("artifacts") - Rake::Task["gitlab:backup:lfs:restore"].invoke unless backup.skipped?("lfs") - Rake::Task["gitlab:shell:setup"].invoke + unless backup.skipped?('db') + unless ENV['force'] == 'yes' + warning = warning = <<-MSG.strip_heredoc + Before restoring the database we recommend removing all existing + tables to avoid future upgrade problems. Be aware that if you have + custom tables in the GitLab database these tables and all data will be + removed. + MSG + ask_to_continue + puts 'Removing all tables. Press `Ctrl-C` within 5 seconds to abort'.yellow + sleep(5) + end + # Drop all tables Load the schema to ensure we don't have any newer tables + # hanging out from a failed upgrade + $progress.puts 'Cleaning the database ... '.blue + Rake::Task['gitlab:db:drop_tables'].invoke + $progress.puts 'done'.green + Rake::Task['gitlab:backup:db:restore'].invoke + end + Rake::Task['gitlab:backup:repo:restore'].invoke unless backup.skipped?('repositories') + Rake::Task['gitlab:backup:uploads:restore'].invoke unless backup.skipped?('uploads') + Rake::Task['gitlab:backup:builds:restore'].invoke unless backup.skipped?('builds') + Rake::Task['gitlab:backup:artifacts:restore'].invoke unless backup.skipped?('artifacts') + Rake::Task['gitlab:backup:lfs:restore'].invoke unless backup.skipped?('lfs') + Rake::Task['gitlab:shell:setup'].invoke backup.cleanup end diff --git a/lib/tasks/gitlab/db.rake b/lib/tasks/gitlab/db.rake new file mode 100644 index 00000000000..4921c6e0bcf --- /dev/null +++ b/lib/tasks/gitlab/db.rake @@ -0,0 +1,35 @@ +namespace :gitlab do + namespace :db do + desc 'GitLab | Manually insert schema migration version' + task :mark_migration_complete, [:version] => :environment do |_, args| + unless args[:version] + puts "Must specify a migration version as an argument".red + exit 1 + end + + version = args[:version].to_i + if version == 0 + puts "Version '#{args[:version]}' must be a non-zero integer".red + exit 1 + end + + sql = "INSERT INTO schema_migrations (version) VALUES (#{version})" + begin + ActiveRecord::Base.connection.execute(sql) + puts "Successfully marked '#{version}' as complete".green + rescue ActiveRecord::RecordNotUnique + puts "Migration version '#{version}' is already marked complete".yellow + end + end + + desc 'Drop all tables' + task :drop_tables => :environment do + connection = ActiveRecord::Base.connection + tables = connection.tables + tables.delete 'schema_migrations' + # Truncate schema_migrations to ensure migrations re-run + connection.execute('TRUNCATE schema_migrations') + tables.each { |t| connection.execute("DROP TABLE #{t}") } + end + end +end diff --git a/spec/tasks/gitlab/backup_rake_spec.rb b/spec/tasks/gitlab/backup_rake_spec.rb index 63bed2414df..320be9a0b61 100644 --- a/spec/tasks/gitlab/backup_rake_spec.rb +++ b/spec/tasks/gitlab/backup_rake_spec.rb @@ -3,9 +3,10 @@ require 'rake' describe 'gitlab:app namespace rake task' do before :all do - Rake.application.rake_require "tasks/gitlab/task_helpers" - Rake.application.rake_require "tasks/gitlab/backup" - Rake.application.rake_require "tasks/gitlab/shell" + Rake.application.rake_require 'tasks/gitlab/task_helpers' + Rake.application.rake_require 'tasks/gitlab/backup' + Rake.application.rake_require 'tasks/gitlab/shell' + Rake.application.rake_require 'tasks/gitlab/db' # empty task as env is already loaded Rake::Task.define_task :environment end @@ -37,6 +38,7 @@ describe 'gitlab:app namespace rake task' do allow(FileUtils).to receive(:mv).and_return(true) allow(Rake::Task["gitlab:shell:setup"]). to receive(:invoke).and_return(true) + ENV['force'] = 'yes' end let(:gitlab_version) { Gitlab::VERSION } @@ -52,13 +54,14 @@ describe 'gitlab:app namespace rake task' do it 'should invoke restoration on match' do allow(YAML).to receive(:load_file). and_return({ gitlab_version: gitlab_version }) - expect(Rake::Task["gitlab:backup:db:restore"]).to receive(:invoke) - expect(Rake::Task["gitlab:backup:repo:restore"]).to receive(:invoke) - expect(Rake::Task["gitlab:backup:builds:restore"]).to receive(:invoke) - expect(Rake::Task["gitlab:backup:uploads:restore"]).to receive(:invoke) - expect(Rake::Task["gitlab:backup:artifacts:restore"]).to receive(:invoke) - expect(Rake::Task["gitlab:backup:lfs:restore"]).to receive(:invoke) - expect(Rake::Task["gitlab:shell:setup"]).to receive(:invoke) + expect(Rake::Task['gitlab:db:drop_tables']).to receive(:invoke) + expect(Rake::Task['gitlab:backup:db:restore']).to receive(:invoke) + expect(Rake::Task['gitlab:backup:repo:restore']).to receive(:invoke) + expect(Rake::Task['gitlab:backup:builds:restore']).to receive(:invoke) + expect(Rake::Task['gitlab:backup:uploads:restore']).to receive(:invoke) + expect(Rake::Task['gitlab:backup:artifacts:restore']).to receive(:invoke) + expect(Rake::Task['gitlab:backup:lfs:restore']).to receive(:invoke) + expect(Rake::Task['gitlab:shell:setup']).to receive(:invoke) expect { run_rake_task('gitlab:backup:restore') }.not_to raise_error end end @@ -177,17 +180,18 @@ describe 'gitlab:app namespace rake task' do end it 'does not invoke repositories restore' do - allow(Rake::Task["gitlab:shell:setup"]). + allow(Rake::Task['gitlab:shell:setup']). to receive(:invoke).and_return(true) allow($stdout).to receive :write - expect(Rake::Task["gitlab:backup:db:restore"]).to receive :invoke - expect(Rake::Task["gitlab:backup:repo:restore"]).not_to receive :invoke - expect(Rake::Task["gitlab:backup:uploads:restore"]).not_to receive :invoke - expect(Rake::Task["gitlab:backup:builds:restore"]).to receive :invoke - expect(Rake::Task["gitlab:backup:artifacts:restore"]).to receive :invoke - expect(Rake::Task["gitlab:backup:lfs:restore"]).to receive :invoke - expect(Rake::Task["gitlab:shell:setup"]).to receive :invoke + expect(Rake::Task['gitlab:db:drop_tables']).to receive :invoke + expect(Rake::Task['gitlab:backup:db:restore']).to receive :invoke + expect(Rake::Task['gitlab:backup:repo:restore']).not_to receive :invoke + expect(Rake::Task['gitlab:backup:uploads:restore']).not_to receive :invoke + expect(Rake::Task['gitlab:backup:builds:restore']).to receive :invoke + expect(Rake::Task['gitlab:backup:artifacts:restore']).to receive :invoke + expect(Rake::Task['gitlab:backup:lfs:restore']).to receive :invoke + expect(Rake::Task['gitlab:shell:setup']).to receive :invoke expect { run_rake_task('gitlab:backup:restore') }.not_to raise_error end end