Commit graph

12 commits

Author SHA1 Message Date
Yorick Peterse
e951a15cdc
Remove feature flag from BackgroundMigrationWorker
The feature has been running on GitLab.com for a while now, without any
problems. This commit removes the eature flag, and enables the feature
for all users.
2018-08-28 13:49:16 +02:00
Yorick Peterse
91b752dce6
Respond to DB health in background migrations
This changes the BackgroundMigration worker so it checks for the health
of the DB before performing a background migration. This in turn allows
us to reduce the minimum interval, without having to worry about blowing
things up if we schedule too many migrations.

In this setup, the BackgroundMigration worker will reschedule jobs as
long as the database is considered to be in an unhealthy state. Once the
database has recovered, the migration can be performed.

To determine if the database is in a healthy state, we look at the
replication lag of any replication slots defined on the primary. If the
lag is deemed to great (100 MB by default) for too many slots, the
migration is rescheduled for a later point in time.

The health checking code is hidden behind a feature flag, allowing us to
disable it if necessary.
2018-08-06 15:20:36 +02:00
gfyoung
dfbe5ce435 Enable frozen string literals for app/workers/*.rb 2018-06-27 07:23:28 +00:00
Yorick Peterse
7f30bb9c29
Run background migrations with a minimum interval
This adds a minimum interval to BackgroundMigrationWorker, ensuring
background migrations of the same class only run once every 5 minutes.
This prevents a thundering herd problem where scheduled migrations all
run at once due to their delays having been expired (e.g. as the result
of a queue being paused for a long time).

If a job was recently executed it's rescheduled with a delay that equals
the remaining time of the job's lease. This means that if the lease
expires in two minutes we only need to wait two minutes, instead of
five.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/41624
2018-01-05 16:23:25 +01:00
Douwe Maan
1e6ca3c41e Consistently schedule Sidekiq jobs 2017-12-05 11:59:39 +01:00
Douwe Maan
0b15570e49 Add ApplicationWorker and make every worker include it 2017-12-05 11:59:39 +01:00
Grzegorz Bizon
5e2baaf39f Improve exception description in bg migrations 2017-07-07 15:08:15 +02:00
Grzegorz Bizon
939473076a Add description to exception in bg migrations worker 2017-07-07 15:08:15 +02:00
Grzegorz Bizon
01128b130b Use integers to schedule delayed background migrations 2017-07-07 15:08:15 +02:00
Grzegorz Bizon
c2efb8acc6 Use ActiveRecord 5 batching to schedule bg migration 2017-07-07 15:08:15 +02:00
Grzegorz Bizon
945cdf326e Make it possible to schedule bg migrations in bulk 2017-07-07 15:08:15 +02:00
Yorick Peterse
d83ee2bbd1
Add the ability to perform background migrations
Background migrations can be used to perform long running data
migrations without these blocking a deployment procedure.

See MR https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11854 for
more information.
2017-06-12 13:24:04 +02:00