Commit graph

16 commits

Author SHA1 Message Date
Yorick Peterse
97731760d7
Re-organize queues to use for Sidekiq
Dumping too many jobs in the same queue (e.g. the "default" queue) is a
dangerous setup. Jobs that take a long time to process can effectively
block any other work from being performed given there are enough of
these jobs.

Furthermore it becomes harder to monitor the jobs as a single queue
could contain jobs for different workers. In such a setup the only
reliable way of getting counts per job is to iterate over all jobs in a
queue, which is a rather time consuming process.

By using separate queues for various workers we have better control over
throughput, we can add weight to queues, and we can monitor queues
better. Some workers still use the same queue whenever their work is
related. For example, the various CI pipeline workers use the same
"pipeline" queue.

This commit includes a Rails migration that moves Sidekiq jobs from the
old queues to the new ones. This migration also takes care of doing the
inverse if ever needed. This does require downtime as otherwise new jobs
could be scheduled in the old queues after this migration completes.

This commit also includes an RSpec test that blacklists the use of the
"default" queue and ensures cron workers use the "cronjob" queue.

Fixes gitlab-org/gitlab-ce#23370
2016-10-21 18:17:07 +02:00
Jacob Vosmaer
8a6776caa9 Use 'exec' in Unicorn and Sidekiq launch scripts
When running Unicorn or Sidekiq in the foreground this change removes
an intermediate /bin/sh process. This makes process supervision in the
GitLab Development Kit more reliable.

This change does not affect Omnibus-GitLab (because there we do not
use these launch scripts). Installations from source do use the launch
scripts but for the standard GitLab init script this change will not
make a difference.

Custom installations using Upstart or Systemd may be affected however,
because under certain configurations these systems count exactly how
many forks happen during process startup, and we are reducing that
number by one here.
2016-04-28 12:11:08 +02:00
Jacob Vosmaer
d28332cc9e Direct sidekiq boot output to log file
This behavior got lost in the previous commit.
2016-01-15 11:12:49 +01:00
Jacob Vosmaer
66f28577ad Improve bin/ launchers 2016-01-14 15:08:15 +01:00
Valery Sizov
9ce8c867ee Fix mailer queue 2015-12-03 15:10:41 +02:00
Douwe Maan
046b283127 Groundwork for merging CI into CE 2015-08-25 18:42:46 -07:00
Douwe Maan
f3b57ce677 Update init scripts. 2015-08-19 13:58:07 -07:00
Douwe Maan
85b29f99f0 Make sure Sidekiq picks up archive_repo queue in production. 2015-04-01 14:36:17 +02:00
Marin Jankovski
976eb7a955 Revert "Delete mailer queue" 2014-09-10 15:28:58 +02:00
Jacob Vosmaer
59d2f6db75 Wrap arguments for [ -z in double quotes 2014-09-08 14:28:42 +02:00
Jacob Vosmaer
494d15cfde Merge pull request #7258 from presto53/fix_init_scripts_interpreter
change bash to sh in web, background_jobs scripts
2014-09-08 14:22:53 +02:00
Pavel Novitskiy
7dd0646e0d fix bash-ism 2014-07-04 15:46:44 +04:00
Pavel Novitskiy
b37247ac31 change bash to sh 2014-07-04 11:45:20 +04:00
ling.su
9c2b046454 Delete mailer queue because we don't use sidekiq_mailer gem and now the mailer queue doesn't exist any more. 2014-07-04 10:51:44 +08:00
Jacob Vosmaer
8e2752f459 Be more selective when killing stray Sidekiqs
Avoid the background_jobs script killing every process with 'sidekiq' in
its argument string (e.g. 'rake gitlab:sidekiq start') by also catching
the number (2) in 'sidekiq 2.14'.
2014-06-12 15:32:47 +02:00
Marin Jankovski
2341cefd6f Move from script to bin directory. 2014-05-27 17:14:41 +02:00
Renamed from script/background_jobs (Browse further)