2018-11-06 06:19:40 +00:00
# Monitoring GitLab with Prometheus
2017-02-08 10:38:24 +00:00
2018-09-06 16:52:18 +00:00
> **Notes:**
2018-11-06 06:19:40 +00:00
> * Prometheus and the various exporters listed in this page are bundled in the
2018-09-06 16:52:18 +00:00
> Omnibus GitLab package. Check each exporter's documentation for the timeline
> they got added. For installations from source you will have to install them
> yourself. Over subsequent releases additional GitLab metrics will be captured.
2018-11-06 06:19:40 +00:00
> * Prometheus services are on by default with GitLab 9.0.
> * Prometheus and its exporters do not authenticate users, and will be available
2018-09-06 16:52:18 +00:00
> to anyone who can access them.
2017-02-08 10:38:24 +00:00
[Prometheus] is a powerful time-series monitoring service, providing a flexible
platform for monitoring GitLab and other software products.
GitLab provides out of the box monitoring with Prometheus, providing easy
access to high quality time-series monitoring of GitLab services.
## Overview
Prometheus works by periodically connecting to data sources and collecting their
2017-02-08 12:40:11 +00:00
performance metrics via the [various exporters ](#prometheus-exporters ). To view
and work with the monitoring data, you can either
[connect directly to Prometheus ](#viewing-performance-metrics ) or utilize a
dashboard tool like [Grafana].
2017-02-08 10:38:24 +00:00
## Configuring Prometheus
>**Note:**
2017-03-16 18:12:54 +00:00
For installations from source you'll have to install and configure it yourself.
2017-02-08 10:38:24 +00:00
2017-03-16 18:12:54 +00:00
Prometheus and it's exporters are on by default, starting with GitLab 9.0.
Prometheus will run as the `gitlab-prometheus` user and listen on
2018-05-29 08:29:19 +00:00
`http://localhost:9090` . By default Prometheus is only accessible from the GitLab server itself.
Each exporter will be automatically set up as a
2017-03-16 18:12:54 +00:00
monitoring target for Prometheus, unless individually disabled.
2017-03-16 18:02:07 +00:00
To disable Prometheus and all of its exporters, as well as any added in the future:
2017-02-08 10:38:24 +00:00
1. Edit `/etc/gitlab/gitlab.rb`
2017-03-16 22:06:47 +00:00
1. Add or find and uncomment the following line, making sure it's set to `false` :
2017-02-08 10:38:24 +00:00
```ruby
2017-03-16 22:06:47 +00:00
prometheus_monitoring['enable'] = false
2017-02-08 10:38:24 +00:00
```
1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
take effect
2018-11-06 06:19:40 +00:00
### Changing the port and address Prometheus listens on
2017-02-09 09:20:30 +00:00
>**Note:**
The following change was added in [GitLab Omnibus 8.17][1261]. Although possible,
2018-05-29 08:29:19 +00:00
it's not recommended to change the port Prometheus listens
2017-02-09 09:20:30 +00:00
on as this might affect or conflict with other services running on the GitLab
server. Proceed at your own risk.
2018-11-13 05:53:37 +00:00
In order to access Prometheus from outside the GitLab server you will need to
2018-05-29 08:29:19 +00:00
set a FQDN or IP in `prometheus['listen_address']` .
2017-02-09 09:20:30 +00:00
To change the address/port that Prometheus listens on:
1. Edit `/etc/gitlab/gitlab.rb`
2017-02-09 09:28:12 +00:00
1. Add or find and uncomment the following line:
2017-02-09 09:20:30 +00:00
```ruby
prometheus['listen_address'] = 'localhost:9090'
```
Replace `localhost:9090` with the address/port you want Prometheus to
2017-07-10 23:33:08 +00:00
listen on. If you would like to allow access to Prometheus to hosts other
than `localhost` , leave out the host, or use `0.0.0.0` to allow public access:
```ruby
prometheus['listen_address'] = ':9090'
# or
prometheus['listen_address'] = '0.0.0.0:9090'
```
2017-02-09 09:20:30 +00:00
1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
take effect
2018-11-06 06:19:40 +00:00
### Using an external Prometheus server
2018-11-13 05:53:37 +00:00
> **Note:** Prometheus and most exporters do not support authentication. We do not recommend exposing them outside the local network.
2018-11-06 06:19:40 +00:00
2018-11-13 05:53:37 +00:00
A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. External servers are recommended for highly available deployments of GitLab with multiple nodes.
2018-11-06 06:19:40 +00:00
2018-11-13 05:53:37 +00:00
1. Edit `/etc/gitlab/gitlab.rb` .
1. Disable the bundled Prometheus:
```ruby
prometheus['enable'] = false
```
1. Set each bundled service's [exporter ](#bundled-software-metrics ) to listen on a network address, for example:
```ruby
gitlab_monitor['listen_address'] = '0.0.0.0'
gitlab_monitor['listen_port'] = '9168'
gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
node_exporter['listen_address'] = '0.0.0.0:9100'
redis_exporter['listen_address'] = '0.0.0.0:9121'
postgres_exporter['listen_address'] = '0.0.0.0:9187'
```
1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions ](https://prometheus.io/docs/prometheus/latest/installation/ ).
1. Add the Prometheus server IP address to the [monitoring IP whitelist ](https://docs.gitlab.com/ce/administration/monitoring/ip_whitelist.html ). For example:
```ruby
gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1']
```
1. Reconfigure GitLab to apply the changes
1. Edit the Prometheus server's configuration file.
1. Add each node's exporters to the Prometheus server's [scrape target configuration ](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E ). For example, a sample snippet using `static_configs` :
```ruby
scrape_configs:
- job_name: 'gitlab_exporters'
static_configs:
- targets: ['1.1.1.1:9168', '1.1.1.1:9236', '1.1.1.1:9236', '1.1.1.1:9100', '1.1.1.1:9121', '1.1.1.1:9187']
- job_name: 'gitlab_metrics'
metrics_path: /-/metrics
static_configs:
- targets: ['1.1.1.1:443']
```
1. Restart the Prometheus server.
2018-11-06 06:19:40 +00:00
2017-02-08 12:40:11 +00:00
## Viewing performance metrics
2017-02-08 10:38:24 +00:00
2017-03-16 18:02:07 +00:00
You can visit `http://localhost:9090` for the dashboard that Prometheus offers by default.
2017-02-09 08:44:05 +00:00
>**Note:**
If SSL has been enabled on your GitLab instance, you may not be able to access
2018-05-29 08:29:19 +00:00
Prometheus on the same browser as GitLab if using the same FQDN due to [HSTS][hsts]. We plan to
2017-02-09 08:44:05 +00:00
[provide access via GitLab][multi-user-prometheus], but in the interim there are
2018-05-29 08:29:19 +00:00
some workarounds: using a separate FQDN, using server IP, using a separate browser for Prometheus, resetting HSTS, or
2018-11-06 06:19:40 +00:00
having [NGINX proxy it][nginx-custom-config].
2017-02-08 10:38:24 +00:00
The performance data collected by Prometheus can be viewed directly in the
Prometheus console or through a compatible dashboard tool.
The Prometheus interface provides a [flexible query language][prom-query] to work
with the collected data where you can visualize their output.
For a more fully featured dashboard, Grafana can be used and has
[official support for Prometheus][prom-grafana].
2017-02-09 08:44:05 +00:00
Sample Prometheus queries:
2018-09-12 08:23:25 +00:00
- **% Memory available:** `((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) or ((node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes) / node_memory_MemTotal_bytes)) * 100`
- **% CPU utilization:** `1 - avg without (mode,cpu) (rate(node_cpu_seconds_total{mode="idle"}[5m]))`
- **Data transmitted:** `rate(node_network_transmit_bytes_total{device!="lo"}[5m])`
- **Data received:** `rate(node_network_receive_bytes_total{device!="lo"}[5m])`
2017-02-09 08:44:05 +00:00
2018-11-06 06:19:40 +00:00
## GitLab metrics
2017-06-20 02:10:05 +00:00
2018-05-21 10:48:59 +00:00
> Introduced in GitLab 9.3.
2017-06-20 02:10:05 +00:00
GitLab monitors its own internal service metrics, and makes them available at the `/-/metrics` endpoint. Unlike other exporters, this endpoint requires authentication as it is available on the same URL and port as user traffic.
[➔ Read more about the GitLab Metrics. ](gitlab_metrics.md )
2018-11-06 06:19:40 +00:00
## Bundled software metrics
2017-02-08 10:38:24 +00:00
2018-11-13 05:53:37 +00:00
Many of the GitLab dependencies bundled in Omnibus GitLab are preconfigured to
export Prometheus metrics.
2017-02-08 10:38:24 +00:00
### Node exporter
2017-02-08 12:40:11 +00:00
The node exporter allows you to measure various machine resources such as
2017-02-08 10:38:24 +00:00
memory, disk and CPU utilization.
2017-02-08 12:40:11 +00:00
[➔ Read more about the node exporter. ](node_exporter.md )
2017-02-08 10:38:24 +00:00
2017-02-08 13:29:02 +00:00
### Redis exporter
The Redis exporter allows you to measure various Redis metrics.
[➔ Read more about the Redis exporter. ](redis_exporter.md )
2017-02-08 13:40:28 +00:00
### Postgres exporter
The Postgres exporter allows you to measure various PostgreSQL metrics.
[➔ Read more about the Postgres exporter. ](postgres_exporter.md )
2017-02-08 14:44:56 +00:00
### GitLab monitor exporter
2017-06-20 02:10:05 +00:00
The GitLab monitor exporter allows you to measure various GitLab metrics, pulled from Redis and the database.
2017-02-08 14:44:56 +00:00
[➔ Read more about the GitLab monitor exporter. ](gitlab_monitor_exporter.md )
2018-11-06 06:19:40 +00:00
## Configuring Prometheus to monitor Kubernetes
> Introduced in GitLab 9.0.
> Pod monitoring introduced in GitLab 9.4.
If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes and [annotated Pods ](https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config ) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][] to monitor them.
To disable the monitoring of Kubernetes:
1. Edit `/etc/gitlab/gitlab.rb`
1. Add or find and uncomment the following line and set it to `false` :
```ruby
prometheus['monitor_kubernetes'] = false
```
1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
take effect
2017-02-08 10:38:24 +00:00
[grafana]: https://grafana.net
2017-02-09 08:44:05 +00:00
[hsts]: https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
[multi-user-prometheus]: https://gitlab.com/gitlab-org/multi-user-prometheus
2017-02-09 09:24:50 +00:00
[nginx-custom-config]: https://docs.gitlab.com/omnibus/settings/nginx.html#inserting-custom-nginx-settings-into-the-gitlab-server-block
2017-02-08 10:38:24 +00:00
[prometheus]: https://prometheus.io
2017-02-09 08:44:05 +00:00
[prom-exporters]: https://prometheus.io/docs/instrumenting/exporters/
2017-02-08 10:38:24 +00:00
[prom-query]: https://prometheus.io/docs/querying/basics
[prom-grafana]: https://prometheus.io/docs/visualization/grafana/
[scrape-config]: https://prometheus.io/docs/operating/configuration/#%3Cscrape_config%3E
[reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure
2017-02-09 09:20:30 +00:00
[1261]: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1261
2017-03-15 19:41:10 +00:00
[prometheus integration]: ../../../user/project/integrations/prometheus.md
2017-03-20 15:17:57 +00:00
[prometheus-cadvisor-metrics]: https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md