Use more accurate timestamps for InfluxDB.
This changes the timestamp of metrics to be more accurate/unique by using Time#to_f combined with a small random jitter value. This combination hopefully reduces the amount of collisions, though there's no way to fully prevent any from occurring. Fixes gitlab-com/operations#175
This commit is contained in:
parent
4d04e91814
commit
aa7cddc4fc
|
@ -2,6 +2,8 @@ module Gitlab
|
||||||
module Metrics
|
module Metrics
|
||||||
# Class for storing details of a single metric (label, value, etc).
|
# Class for storing details of a single metric (label, value, etc).
|
||||||
class Metric
|
class Metric
|
||||||
|
JITTER_RANGE = 0.000001..0.001
|
||||||
|
|
||||||
attr_reader :series, :values, :tags, :created_at
|
attr_reader :series, :values, :tags, :created_at
|
||||||
|
|
||||||
# series - The name of the series (as a String) to store the metric in.
|
# series - The name of the series (as a String) to store the metric in.
|
||||||
|
@ -16,11 +18,29 @@ module Gitlab
|
||||||
|
|
||||||
# Returns a Hash in a format that can be directly written to InfluxDB.
|
# Returns a Hash in a format that can be directly written to InfluxDB.
|
||||||
def to_hash
|
def to_hash
|
||||||
|
# InfluxDB overwrites an existing point if a new point has the same
|
||||||
|
# series, tag set, and timestamp. In a highly concurrent environment
|
||||||
|
# this means that using the number of seconds since the Unix epoch is
|
||||||
|
# inevitably going to collide with another timestamp. For example, two
|
||||||
|
# Rails requests processed by different processes may end up generating
|
||||||
|
# metrics using the _exact_ same timestamp (in seconds).
|
||||||
|
#
|
||||||
|
# Due to the way InfluxDB is set up there's no solution to this problem,
|
||||||
|
# all we can do is lower the amount of collisions. We do this by using
|
||||||
|
# Time#to_f which returns the seconds as a Float providing greater
|
||||||
|
# accuracy. We then add a small random value that is large enough to
|
||||||
|
# distinguish most timestamps but small enough to not alter the amount
|
||||||
|
# of seconds.
|
||||||
|
#
|
||||||
|
# See https://gitlab.com/gitlab-com/operations/issues/175 for more
|
||||||
|
# information.
|
||||||
|
time = @created_at.to_f + rand(JITTER_RANGE)
|
||||||
|
|
||||||
{
|
{
|
||||||
series: @series,
|
series: @series,
|
||||||
tags: @tags,
|
tags: @tags,
|
||||||
values: @values,
|
values: @values,
|
||||||
timestamp: @created_at.to_i * 1_000_000_000
|
timestamp: (time * 1_000_000_000).to_i
|
||||||
}
|
}
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
Loading…
Reference in New Issue