Commit graph

12 commits

Author SHA1 Message Date
Yorick Peterse
91b752dce6
Respond to DB health in background migrations
This changes the BackgroundMigration worker so it checks for the health
of the DB before performing a background migration. This in turn allows
us to reduce the minimum interval, without having to worry about blowing
things up if we schedule too many migrations.

In this setup, the BackgroundMigration worker will reschedule jobs as
long as the database is considered to be in an unhealthy state. Once the
database has recovered, the migration can be performed.

To determine if the database is in a healthy state, we look at the
replication lag of any replication slots defined on the primary. If the
lag is deemed to great (100 MB by default) for too many slots, the
migration is rescheduled for a later point in time.

The health checking code is hidden behind a feature flag, allowing us to
disable it if necessary.
2018-08-06 15:20:36 +02:00
Grzegorz Bizon
2930c0e3d0 Remove obsolete argument from bg migrations code 2017-07-18 10:43:12 +02:00
Grzegorz Bizon
e41d42d6a2 Simplify background migrations stealing code
Simply re-raise an exception when it occurs, but guarantee that no
background migration is lost in the process.
2017-07-17 14:02:12 +02:00
Grzegorz Bizon
af41bd41e9 Fix off-by-one error in background migration retries 2017-07-17 10:45:52 +02:00
Grzegorz Bizon
7b146ab6c3 Recover from all exceptions when stealing bg migration
It also makes it possible to gracefully retry a migration in order to
avoid problems like deadlocks.
2017-07-17 10:16:56 +02:00
Grzegorz Bizon
01c55ffca8 Catch exceptions when stealing background migrations 2017-07-14 15:40:51 +02:00
Grzegorz Bizon
39b96f02dc Avoid race condition when stealing a background migration
We first pop a job from the Sidekiq queue / scheduled set and only if
this has been successfully deleted we process the job. This makes it
possible to minimize a possibility of a race condition happening.
2017-07-14 12:55:52 +02:00
Grzegorz Bizon
beffbc8aa2 Fix mocks in background migrations specs 2017-07-13 12:08:03 +02:00
Grzegorz Bizon
2a0ead2c47 Implement draining scheduled sets of background migrations 2017-07-13 11:44:52 +02:00
Grzegorz Bizon
388abbd10c Extract background migratons queue class method 2017-07-11 15:42:00 +02:00
Grzegorz Bizon
0430b76441 Enable Style/DotPosition Rubocop 👮 2017-06-21 13:48:12 +00:00
Yorick Peterse
d83ee2bbd1
Add the ability to perform background migrations
Background migrations can be used to perform long running data
migrations without these blocking a deployment procedure.

See MR https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11854 for
more information.
2017-06-12 13:24:04 +02:00