Fix metric files being wiped after the app starts
When we hit our app with the initial request, in `warmup`, some metrics already being created as well as corresponding files. If we do `multiproc_file_dir` cleanup after that, we delete the files from the dir while keeping them in memory which leads to the incorrect behavior: the metric is being updated in in-memory, while is not present in the db, not sent to Prometheus as the result.
This commit is contained in:
parent
7a48b4d6b0
commit
3f9815865c
|
@ -0,0 +1,5 @@
|
|||
---
|
||||
title: Fix active metric files being wiped after the app starts
|
||||
merge_request: 31668
|
||||
author:
|
||||
type: fixed
|
15
config.ru
15
config.ru
|
@ -17,7 +17,22 @@ end
|
|||
|
||||
require ::File.expand_path('../config/environment', __FILE__)
|
||||
|
||||
|
||||
# The following is necessary to ensure stale Prometheus metrics don't accumulate over time.
|
||||
# It needs to be done as early as here to ensure metrics files aren't deleted.
|
||||
# After we hit our app in `warmup`, first metrics and corresponding files already being created,
|
||||
# for example in `lib/gitlab/metrics/requests_rack_middleware.rb`.
|
||||
def cleanup_prometheus_multiproc_dir
|
||||
if dir = ::Prometheus::Client.configuration.multiprocess_files_dir
|
||||
old_metrics = Dir[File.join(dir, '*.db')]
|
||||
|
||||
FileUtils.rm_rf(old_metrics)
|
||||
end
|
||||
end
|
||||
|
||||
warmup do |app|
|
||||
cleanup_prometheus_multiproc_dir
|
||||
|
||||
client = Rack::MockRequest.new(app)
|
||||
client.get('/')
|
||||
end
|
||||
|
|
|
@ -51,22 +51,3 @@ if !Rails.env.test? && Gitlab::Metrics.prometheus_metrics_enabled?
|
|||
end
|
||||
end
|
||||
end
|
||||
|
||||
def cleanup_prometheus_multiproc_dir
|
||||
# The following is necessary to ensure stale Prometheus metrics don't
|
||||
# accumulate over time. It needs to be done in this hook as opposed to
|
||||
# inside an init script to ensure metrics files aren't deleted after new
|
||||
# unicorn workers start after a SIGUSR2 is received.
|
||||
if dir = ::Prometheus::Client.configuration.multiprocess_files_dir
|
||||
old_metrics = Dir[File.join(dir, '*.db')]
|
||||
FileUtils.rm_rf(old_metrics)
|
||||
end
|
||||
end
|
||||
|
||||
Gitlab::Cluster::LifecycleEvents.on_master_start do
|
||||
cleanup_prometheus_multiproc_dir
|
||||
end
|
||||
|
||||
Gitlab::Cluster::LifecycleEvents.on_master_restart do
|
||||
cleanup_prometheus_multiproc_dir
|
||||
end
|
||||
|
|
Loading…
Reference in New Issue