The Correlation ID is taken or generated from received X-Request-ID.
Then it is being passed to all executed services (sidekiq workers
or gitaly calls).
The Correlation ID is logged in all structured logs as `correlation_id`.
The sidekiq memory killer prints warnings like
gitlab-sidekiq[8245]: 2017-09-27T11:31:43.192Z 8245 TID-ov9l6b4e0 WARN:
this thread will shut down PID 8245 - Worker ProjectWebHookWorker -
JID-dd3b1c23a653deec5e575ab4in 900 seconds
There is a space missing between the JID and 'in'.
Log sidekiq arguments as json
Logging the sidekiq job arguments as a ruby literal structure
is akward. Specially when parsing the logs.
JSON is a standar format.
See merge request !3735
This makes the memory growth-triggered Sidekiq restarts more reliable by
reducing the chance that Sidekiq ends up in a state where it is not
accepting new jobs but also not shutting down: SIGKILL is more likely to
work than SIGTERM.
It looks like SIGTERM may not be enough to shut down a Sidekiq process
when its RSS has gotten too big. This change will allow us to experiment
with sending SIGKILL instead of SIGTERM to Sidekiq processes on
gitlab.com.
When enabled, this middleware allows Sidekiq to detect that its RSS has
exceeded a maximum value, triggering a graceful shutdown. This
middleware should be combined with external process supervision that
will restart Sidekiq after the graceful shutdown, such as Runit.
Kamil Trzciński