Make usage ping scheduling more robust
On GitLab.com, we see that cron jobs sometimes don't run. This appears to be because the process that polls for current cron jobs only has a 60 second validity period, so if (for whatever reason) it misses those 60 seconds, we have to wait until the next execution. For the usage ping, this is particularly problematic, as that's only scheduled to run once a week. Changing it to run for every minute in a 10 minute period should work around this, if the above diagnosis is correct. The job itself obtains an exclusive lease for 24 hours, so rescheduling in quick succession is safe.
This commit is contained in:
parent
5843a43c16
commit
f4e94c6ff7
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
title: Make usage ping scheduling more robust
|
||||
merge_request:
|
||||
author:
|
||||
type: fixed
|
|
@ -113,12 +113,14 @@ class Settings < Settingslogic
|
|||
URI.parse(url_without_path).host
|
||||
end
|
||||
|
||||
# Random cron time every Sunday to load balance usage pings
|
||||
def cron_random_weekly_time
|
||||
# Runs every minute in a random ten-minute period on Sundays, to balance the
|
||||
# load on the server receiving these pings. The usage ping is safe to run
|
||||
# multiple times because of a 24 hour exclusive lock.
|
||||
def cron_for_usage_ping
|
||||
hour = rand(24)
|
||||
minute = rand(60)
|
||||
minute = rand(6)
|
||||
|
||||
"#{minute} #{hour} * * 0"
|
||||
"#{minute}0-#{minute}9 #{hour} * * 0"
|
||||
end
|
||||
end
|
||||
end
|
||||
|
@ -398,7 +400,7 @@ Settings.cron_jobs['stuck_import_jobs_worker'] ||= Settingslogic.new({})
|
|||
Settings.cron_jobs['stuck_import_jobs_worker']['cron'] ||= '15 * * * *'
|
||||
Settings.cron_jobs['stuck_import_jobs_worker']['job_class'] = 'StuckImportJobsWorker'
|
||||
Settings.cron_jobs['gitlab_usage_ping_worker'] ||= Settingslogic.new({})
|
||||
Settings.cron_jobs['gitlab_usage_ping_worker']['cron'] ||= Settings.__send__(:cron_random_weekly_time)
|
||||
Settings.cron_jobs['gitlab_usage_ping_worker']['cron'] ||= Settings.__send__(:cron_for_usage_ping)
|
||||
Settings.cron_jobs['gitlab_usage_ping_worker']['job_class'] = 'GitlabUsagePingWorker'
|
||||
|
||||
Settings.cron_jobs['schedule_update_user_activity_worker'] ||= Settingslogic.new({})
|
||||
|
|
Loading…
Reference in New Issue