Commit graph

12 commits

Author SHA1 Message Date
Zeger-Jan van de Weg
65840591cd
Gitaly metrics check for read/writeability
Prior to this change, health checks checked for writeability of the NFS
shards. Given we're moving away from that, this patch extends the checks
for Gitaly to check for read and writeability.

Potentially some dashboards will break, as over time these metrics will
no longer appear as Prometheus doesn't get the data anymore.
Observability in the circuit breaker will be reduced, but its not
expected to be turned on and the circuit breaker is being removed soon
too.

Closes https://gitlab.com/gitlab-org/gitaly/issues/1218
2018-06-27 08:56:19 +02:00
Mario de la Ossa
eaada9d706 use Gitlab::UserSettings directly as a singleton instead of including/extending it 2018-02-02 18:39:55 +00:00
Bob Van Landuyt
f1ae1e39ce Move the circuitbreaker check out in a separate process
Moving the check out of the general requests, makes sure we don't have
any slowdown in the regular requests.

To keep the process performing this checks small, the check is still
performed inside a unicorn. But that is called from a process running
on the same server.

Because the checks are now done outside normal request, we can have a
simpler failure strategy:

The check is now performed in the background every
`circuitbreaker_check_interval`. Failures are logged in redis. The
failures are reset when the check succeeds. Per check we will try
`circuitbreaker_access_retries` times within
`circuitbreaker_storage_timeout` seconds.

When the number of failures exceeds
`circuitbreaker_failure_count_threshold`, we will block access to the
storage.

After `failure_reset_time` of no checks, we will clear the stored
failures. This could happen when the process that performs the checks
is not running.
2017-12-08 09:11:39 +01:00
Nick Thomas
b00aaeced7 Fix the filesystem shard health check to check all configured shards 2017-09-19 11:05:42 +01:00
Sean McGivern
25d241ae97 Merge branch '33949-remove-healthcheck-access-token' into 'master'
Remove the need to use health check token by adding ability to whitelist hosts

Closes #33949

See merge request !12612
2017-07-11 08:51:27 +00:00
Paul Charlton
cb3b4a15e6 Support multiple Redis instances based on queue type 2017-07-11 03:35:47 +00:00
Pawel Chojnacki
063f03b9d7 Differentiate shared test context using options hash instead of subject.
+ fix typos, and capitalization
+ point configuration to `gitlab.rb` as well
2017-07-10 17:00:43 +02:00
Pawel Chojnacki
2951a09967 Add tests for token auth. 2017-07-06 15:43:57 +02:00
Pawel Chojnacki
6d60e265b1 Add spaces between test phases
+ fix wrong test setup
2017-07-05 00:46:12 +02:00
Pawel Chojnacki
18521584bd Remove the need to use health check token
in favor of whitelist that will be used to
control the access to monitoring resources
2017-07-05 00:46:11 +02:00
Pawel Chojnacki
0f4050430d Split metrics from health controller into metrics controller 2017-06-02 19:45:58 +02:00
Paweł Chojnacki
c3e43c9ba3 Add /-/readiness /-/liveness and /-/health_metrics endpoints to track application readiness 2017-04-07 10:27:15 +00:00