Reload the schema before restoring a database backup

CHANGELOG

@@ -24,6 +24,7 @@ v 8.6.0 (unreleased)
   - Don't load all of GitLab in mail_room
   - HTTP error pages work independently from location and config (Artem Sidorenko)
   - Update `omniauth-saml` to 1.5.0 to allow for custom response attributes to be set
+  - Add option to reload the schema before restoring a database backup. !2807
   - Memoize @group in Admin::GroupsController (Yatish Mehta)
   - Indicate how much an MR diverged from the target branch (Pierre de La Morinerie)
   - Added omniauth-auth0 Gem (Daniel Carraro)

doc/raketasks/backup_restore.md

@@ -249,6 +249,9 @@ reconfigure` after changing `gitlab-secrets.json`.
 ### Installation from source
 
 ```
+# Stop processes that are connected to the database
+sudo service gitlab stop
+
 bundle exec rake gitlab:backup:restore RAILS_ENV=production
 ```
 

doc/update/README.md

@@ -15,3 +15,4 @@ Depending on the installation method and your GitLab version, there are multiple
 
 - [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostgreSQL.
 - [MySQL installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/database_mysql.md) contains additional information about configuring GitLab to work with a MySQL database.
+- [Restoring from backup after a failed upgrade](restore_after_failure.md)

doc/update/restore_after_failure.md

@@ -0,0 +1,83 @@
+# Restoring from backup after a failed upgrade
+
+Upgrades are usually smooth and restoring from backup is a rare occurrence.
+However, it's important to know how to recover when problems do arise.
+
+## Roll back to an earlier version and restore a backup
+
+In some cases after a failed upgrade, the fastest solution is to roll back to
+the previous version you were using.
+
+First, roll back the code or package. For source installations this involves
+checking out the older version (branch or tag). For Omnibus installations this
+means installing the older .deb or .rpm package. Then, restore from a backup.
+Follow the instructions in the
+[Backup and Restore](../raketasks/backup_restore.md#restore-a-previously-created-backup)
+documentation.
+
+## Potential problems on the next upgrade
+
+When a rollback is necessary it can produce problems on subsequent upgrade
+attempts. This is because some tables may have been added during the failed
+upgrade. If these tables are still present after you restore from the
+older backup it can lead to migration failures on future upgrades.
+
+Starting in GitLab 8.6 we drop all tables prior to importing the backup to
+prevent this problem. If you've restored a backup to a version prior to 8.6 you
+may need to manually correct the problem next time you upgrade.
+
+Example error:
+
+```
+== 20151103134857 CreateLfsObjects: migrating =================================
+-- create_table(:lfs_objects)
+rake aborted!
+StandardError: An error has occurred, this and all later migrations canceled:
+
+PG::DuplicateTable: ERROR:  relation "lfs_objects" already exists
+```
+
+Copy the version from the error. In this case the version number is
+`20151103134857`.
+
+>**WARNING:** Use the following steps only if you are certain this is what you
+need to do.
+
+### GitLab 8.6+
+
+Pass the version to a database rake task to manually mark the migration as
+complete.
+
+```
+# Source install
+sudo -u git -H bundle exec rake gitlab:db:mark_migration_complete[20151103134857] RAILS_ENV=production
+
+# Omnibus install
+sudo gitlab-rake gitlab:db:mark_migration_complete[20151103134857]
+```
+
+Once the migration is successfully marked, run the rake `db:migrate` task again.
+You will likely have to repeat this process several times until all failed
+migrations are marked complete.
+
+### GitLab < 8.6
+
+```
+# Source install
+sudo -u git -H bundle exec rails console production
+
+# Omnibus install
+sudo gitlab-rails console
+```
+
+At the Rails console, type the following commands:
+
+```
+ActiveRecord::Base.connection.execute("INSERT INTO schema_migrations (version) VALUES('20151103134857')")
+exit
+```
+
+Once the migration is successfully marked, run the rake `db:migrate` task again.
+You will likely have to repeat this process several times until all failed
+migrations are marked complete.
+

lib/tasks/gitlab/backup.rake

@@ -22,7 +22,7 @@ namespace :gitlab do
     end
 
     # Restore backup of GitLab system
-    desc "GitLab | Restore a previously created backup"
+    desc 'GitLab | Restore a previously created backup'
     task restore: :environment do
       warn_user_is_not_gitlab
       configure_cron_mode
@@ -30,13 +30,31 @@ namespace :gitlab do
       backup = Backup::Manager.new
       backup.unpack
 
-      Rake::Task["gitlab:backup:db:restore"].invoke unless backup.skipped?("db")
-      Rake::Task["gitlab:backup:repo:restore"].invoke unless backup.skipped?("repositories")
-      Rake::Task["gitlab:backup:uploads:restore"].invoke unless backup.skipped?("uploads")
-      Rake::Task["gitlab:backup:builds:restore"].invoke unless backup.skipped?("builds")
-      Rake::Task["gitlab:backup:artifacts:restore"].invoke unless backup.skipped?("artifacts")
-      Rake::Task["gitlab:backup:lfs:restore"].invoke unless backup.skipped?("lfs")
-      Rake::Task["gitlab:shell:setup"].invoke
+      unless backup.skipped?('db')
+        unless ENV['force'] == 'yes'
+          warning = warning = <<-MSG.strip_heredoc
+            Before restoring the database we recommend removing all existing
+            tables to avoid future upgrade problems. Be aware that if you have
+            custom tables in the GitLab database these tables and all data will be
+            removed.
+          MSG
+          ask_to_continue
+          puts 'Removing all tables. Press `Ctrl-C` within 5 seconds to abort'.yellow
+          sleep(5)
+        end
+        # Drop all tables Load the schema to ensure we don't have any newer tables
+        # hanging out from a failed upgrade
+        $progress.puts 'Cleaning the database ... '.blue
+        Rake::Task['gitlab:db:drop_tables'].invoke
+        $progress.puts 'done'.green
+        Rake::Task['gitlab:backup:db:restore'].invoke
+      end
+      Rake::Task['gitlab:backup:repo:restore'].invoke unless backup.skipped?('repositories')
+      Rake::Task['gitlab:backup:uploads:restore'].invoke unless backup.skipped?('uploads')
+      Rake::Task['gitlab:backup:builds:restore'].invoke unless backup.skipped?('builds')
+      Rake::Task['gitlab:backup:artifacts:restore'].invoke unless backup.skipped?('artifacts')
+      Rake::Task['gitlab:backup:lfs:restore'].invoke unless backup.skipped?('lfs')
+      Rake::Task['gitlab:shell:setup'].invoke
 
       backup.cleanup
     end

lib/tasks/gitlab/db.rake

@@ -0,0 +1,35 @@
+namespace :gitlab do
+  namespace :db do
+    desc 'GitLab | Manually insert schema migration version'
+    task :mark_migration_complete, [:version] => :environment do |_, args|
+      unless args[:version]
+        puts "Must specify a migration version as an argument".red
+        exit 1
+      end
+
+      version = args[:version].to_i
+      if version == 0
+        puts "Version '#{args[:version]}' must be a non-zero integer".red
+        exit 1
+      end
+
+      sql = "INSERT INTO schema_migrations (version) VALUES (#{version})"
+      begin
+        ActiveRecord::Base.connection.execute(sql)
+        puts "Successfully marked '#{version}' as complete".green
+      rescue ActiveRecord::RecordNotUnique
+        puts "Migration version '#{version}' is already marked complete".yellow
+      end
+    end
+
+    desc 'Drop all tables'
+    task :drop_tables => :environment do
+      connection = ActiveRecord::Base.connection
+      tables = connection.tables
+      tables.delete 'schema_migrations'
+      # Truncate schema_migrations to ensure migrations re-run
+      connection.execute('TRUNCATE schema_migrations')
+      tables.each { |t| connection.execute("DROP TABLE #{t}") }
+    end
+  end
+end

spec/tasks/gitlab/backup_rake_spec.rb

@@ -3,9 +3,10 @@ require 'rake'
 
 describe 'gitlab:app namespace rake task' do
   before :all do
-    Rake.application.rake_require "tasks/gitlab/task_helpers"
-    Rake.application.rake_require "tasks/gitlab/backup"
-    Rake.application.rake_require "tasks/gitlab/shell"
+    Rake.application.rake_require 'tasks/gitlab/task_helpers'
+    Rake.application.rake_require 'tasks/gitlab/backup'
+    Rake.application.rake_require 'tasks/gitlab/shell'
+    Rake.application.rake_require 'tasks/gitlab/db'
     # empty task as env is already loaded
     Rake::Task.define_task :environment
   end
@@ -37,6 +38,7 @@ describe 'gitlab:app namespace rake task' do
         allow(FileUtils).to receive(:mv).and_return(true)
         allow(Rake::Task["gitlab:shell:setup"]).
           to receive(:invoke).and_return(true)
+        ENV['force'] = 'yes'
       end
 
       let(:gitlab_version) { Gitlab::VERSION }
@@ -52,13 +54,14 @@ describe 'gitlab:app namespace rake task' do
       it 'should invoke restoration on match' do
         allow(YAML).to receive(:load_file).
           and_return({ gitlab_version: gitlab_version })
-        expect(Rake::Task["gitlab:backup:db:restore"]).to receive(:invoke)
-        expect(Rake::Task["gitlab:backup:repo:restore"]).to receive(:invoke)
-        expect(Rake::Task["gitlab:backup:builds:restore"]).to receive(:invoke)
-        expect(Rake::Task["gitlab:backup:uploads:restore"]).to receive(:invoke)
-        expect(Rake::Task["gitlab:backup:artifacts:restore"]).to receive(:invoke)
-        expect(Rake::Task["gitlab:backup:lfs:restore"]).to receive(:invoke)
-        expect(Rake::Task["gitlab:shell:setup"]).to receive(:invoke)
+        expect(Rake::Task['gitlab:db:drop_tables']).to receive(:invoke)
+        expect(Rake::Task['gitlab:backup:db:restore']).to receive(:invoke)
+        expect(Rake::Task['gitlab:backup:repo:restore']).to receive(:invoke)
+        expect(Rake::Task['gitlab:backup:builds:restore']).to receive(:invoke)
+        expect(Rake::Task['gitlab:backup:uploads:restore']).to receive(:invoke)
+        expect(Rake::Task['gitlab:backup:artifacts:restore']).to receive(:invoke)
+        expect(Rake::Task['gitlab:backup:lfs:restore']).to receive(:invoke)
+        expect(Rake::Task['gitlab:shell:setup']).to receive(:invoke)
         expect { run_rake_task('gitlab:backup:restore') }.not_to raise_error
       end
     end
@@ -177,17 +180,18 @@ describe 'gitlab:app namespace rake task' do
     end
 
     it 'does not invoke repositories restore' do
-      allow(Rake::Task["gitlab:shell:setup"]).
+      allow(Rake::Task['gitlab:shell:setup']).
         to receive(:invoke).and_return(true)
       allow($stdout).to receive :write
 
-      expect(Rake::Task["gitlab:backup:db:restore"]).to receive :invoke
-      expect(Rake::Task["gitlab:backup:repo:restore"]).not_to receive :invoke
-      expect(Rake::Task["gitlab:backup:uploads:restore"]).not_to receive :invoke
-      expect(Rake::Task["gitlab:backup:builds:restore"]).to receive :invoke
-      expect(Rake::Task["gitlab:backup:artifacts:restore"]).to receive :invoke
-      expect(Rake::Task["gitlab:backup:lfs:restore"]).to receive :invoke
-      expect(Rake::Task["gitlab:shell:setup"]).to receive :invoke
+      expect(Rake::Task['gitlab:db:drop_tables']).to receive :invoke
+      expect(Rake::Task['gitlab:backup:db:restore']).to receive :invoke
+      expect(Rake::Task['gitlab:backup:repo:restore']).not_to receive :invoke
+      expect(Rake::Task['gitlab:backup:uploads:restore']).not_to receive :invoke
+      expect(Rake::Task['gitlab:backup:builds:restore']).to receive :invoke
+      expect(Rake::Task['gitlab:backup:artifacts:restore']).to receive :invoke
+      expect(Rake::Task['gitlab:backup:lfs:restore']).to receive :invoke
+      expect(Rake::Task['gitlab:shell:setup']).to receive :invoke
       expect { run_rake_task('gitlab:backup:restore') }.not_to raise_error
     end
   end