Commit graph

16 commits

Author SHA1 Message Date
Douwe Maan
ccb5bad6b8 Send SIGSTP before SIGTERM to actually give Sidekiq jobs 30s to finish when the memory killer kicks in 2017-11-02 15:33:19 +01:00
Benjamin Drung
74846409ce Sidekiq::MemoryKiller: Add missing space in log message
The sidekiq memory killer prints warnings like

gitlab-sidekiq[8245]: 2017-09-27T11:31:43.192Z 8245 TID-ov9l6b4e0 WARN:
this thread will shut down PID 8245 - Worker ProjectWebHookWorker -
JID-dd3b1c23a653deec5e575ab4in 900 seconds

There is a space missing between the JID and 'in'.
2017-10-04 10:50:43 +02:00
Rémy Coutable
8d333c87f1 Merge branch 'aiionx_sidekiq_log_patch' into 'master'
Log sidekiq arguments as json

Logging the sidekiq job arguments as a ruby literal structure 
is akward. Specially when parsing the logs.

JSON is a standar format.



See merge request !3735
2016-10-03 14:20:18 +00:00
Stan Hu
f4aac77389 Add support for using RequestStore within Sidekiq tasks via SIDEKIQ_REQUEST_STORE env variable
This significantly reduces the DB churn in the PostReceive task when it
performs reference extraction.

See #18663
2016-07-25 17:59:09 -07:00
James Lopez
95f630daeb even more debug 2016-07-06 10:29:31 +02:00
James Lopez
a7b1b51226 better debugging for memory killer middleware 2016-06-27 12:11:01 +02:00
Jacob Vosmaer
103c405339 Use SIGTERM during Sidekiq memory shutdown
This makes the memory killer behave more like 'sidekiqctl stop'.
2016-04-21 14:49:15 +02:00
David
dc20dd275f aiionx_sidekiq_log_patch 2016-04-15 02:18:46 +00:00
Jacob Vosmaer
c291ff9c6f Use SIGKILL by default in Sidekiq::MemoryKiller
This makes the memory growth-triggered Sidekiq restarts more reliable by
reducing the chance that Sidekiq ends up in a state where it is not
accepting new jobs but also not shutting down: SIGKILL is more likely to
work than SIGTERM.
2015-05-13 17:09:12 +02:00
Jacob Vosmaer
1c1f18b416 Add SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL env var
It looks like SIGTERM may not be enough to shut down a Sidekiq process
when its RSS has gotten too big. This change will allow us to experiment
with sending SIGKILL instead of SIGTERM to Sidekiq processes on
gitlab.com.
2015-05-07 18:47:03 +02:00
Jacob Vosmaer
2fcef3278c Fix typo 2014-12-08 13:39:18 +01:00
Jacob Vosmaer
3dd86b83ba Use constants instead of getters 2014-12-08 13:19:31 +01:00
Jacob Vosmaer
4f9a14061b Wait 15 minutes before Sidekiq MemoryKiller action 2014-12-05 17:29:34 +01:00
Jacob Vosmaer
d336127a20 Add comments to the MemoryKiller middleware 2014-11-28 15:19:03 +01:00
Jacob Vosmaer
64ab6c9ed5 Add 'MemoryKiller' Sidekiq middleware
When enabled, this middleware allows Sidekiq to detect that its RSS has
exceeded a maximum value, triggering a graceful shutdown. This
middleware should be combined with external process supervision that
will restart Sidekiq after the graceful shutdown, such as Runit.
2014-11-28 15:01:41 +01:00
Jacob Vosmaer
03ae6b89bd Log Sidekiq arguments 2014-07-28 16:46:36 +02:00