91b752dce6
This changes the BackgroundMigration worker so it checks for the health of the DB before performing a background migration. This in turn allows us to reduce the minimum interval, without having to worry about blowing things up if we schedule too many migrations. In this setup, the BackgroundMigration worker will reschedule jobs as long as the database is considered to be in an unhealthy state. Once the database has recovered, the migration can be performed. To determine if the database is in a healthy state, we look at the replication lag of any replication slots defined on the primary. If the lag is deemed to great (100 MB by default) for too many slots, the migration is rescheduled for a later point in time. The health checking code is hidden behind a feature flag, allowing us to disable it if necessary.
56 lines
1.8 KiB
Ruby
56 lines
1.8 KiB
Ruby
module Gitlab
|
|
module BackgroundMigration
|
|
def self.queue
|
|
@queue ||= BackgroundMigrationWorker.sidekiq_options['queue']
|
|
end
|
|
|
|
# Begins stealing jobs from the background migrations queue, blocking the
|
|
# caller until all jobs have been completed.
|
|
#
|
|
# When a migration raises a StandardError is is going to be retries up to
|
|
# three times, for example, to recover from a deadlock.
|
|
#
|
|
# When Exception is being raised, it enqueues the migration again, and
|
|
# re-raises the exception.
|
|
#
|
|
# steal_class - The name of the class for which to steal jobs.
|
|
def self.steal(steal_class)
|
|
enqueued = Sidekiq::Queue.new(self.queue)
|
|
scheduled = Sidekiq::ScheduledSet.new
|
|
|
|
[scheduled, enqueued].each do |queue|
|
|
queue.each do |job|
|
|
migration_class, migration_args = job.args
|
|
|
|
next unless job.queue == self.queue
|
|
next unless migration_class == steal_class
|
|
|
|
begin
|
|
perform(migration_class, migration_args) if job.delete
|
|
rescue Exception # rubocop:disable Lint/RescueException
|
|
BackgroundMigrationWorker # enqueue this migration again
|
|
.perform_async(migration_class, migration_args)
|
|
|
|
raise
|
|
end
|
|
end
|
|
end
|
|
end
|
|
|
|
##
|
|
# Performs a background migration.
|
|
#
|
|
# class_name - The name of the background migration class as defined in the
|
|
# Gitlab::BackgroundMigration namespace.
|
|
#
|
|
# arguments - The arguments to pass to the background migration's "perform"
|
|
# method.
|
|
def self.perform(class_name, arguments)
|
|
migration_class_for(class_name).new.perform(*arguments)
|
|
end
|
|
|
|
def self.migration_class_for(class_name)
|
|
const_get(class_name)
|
|
end
|
|
end
|
|
end
|