gitlab-org--gitlab-foss/doc/administration/high_availability/monitoring_node.md

type
reference

Configuring a Monitoring node for Scaling and High Availability

Introduced in GitLab 12.0.

You can configure a Prometheus node to monitor GitLab.

Standalone Monitoring node using Omnibus GitLab

The Omnibus GitLab package can be used to configure a standalone Monitoring node running Prometheus and Grafana. The monitoring node is not highly available. See Scaling and High Availability for an overview of GitLab scaling and high availability options.

The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with Omnibus:

1. SSH into the Monitoring node.

2. Download/install the Omnibus GitLab package you want using steps 1 and 2 from the GitLab downloads page.

   • Do not complete any other steps on the download page.

3. Make sure to collect CONSUL_SERVER_NODES, which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z

4. Edit /etc/gitlab/gitlab.rb and add the contents:

   external_url 'http://gitlab.example.com'
   
   # Enable Prometheus
   prometheus['enable'] = true
   prometheus['listen_address'] = '0.0.0.0:9090'
   prometheus['monitor_kubernetes'] = false
   
   # Enable Login form
   grafana['disable_login_form'] = false
   
   # Enable Grafana
   grafana['enable'] = true
   grafana['admin_password'] = 'toomanysecrets'
   
   # Enable service discovery for Prometheus
   consul['enable'] = true
   consul['monitoring_service_discovery'] =  true
   
   # Replace placeholders
   # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
   # with the addresses of the Consul server nodes
   consul['configuration'] = {
      retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
   }
   
   # Disable all other services
   gitlab_rails['auto_migrate'] = false
   alertmanager['enable'] = false
   gitaly['enable'] = false
   gitlab_exporter['enable'] = false
   gitlab_workhorse['enable'] = false
   nginx['enable'] = true
   postgres_exporter['enable'] = false
   postgresql['enable'] = false
   redis['enable'] = false
   redis_exporter['enable'] = false
   sidekiq['enable'] = false
   puma['enable'] = false
   node_exporter['enable'] = false
   gitlab_exporter['enable'] = false
   

5. Run sudo gitlab-ctl reconfigure to compile the configuration.

The next step is to tell all the other nodes where the monitoring node is:

1. Edit /etc/gitlab/gitlab.rb, and add, or find and uncomment the following line:

   gitlab_rails['prometheus_address'] = '10.0.0.1:9090'
   

   Where 10.0.0.1:9090 is the IP address and port of the Prometheus node.

2. Save the file and reconfigure GitLab for the changes to take effect.

Migrating to Service Discovery

Once monitoring using Service Discovery is enabled with consul['monitoring_service_discovery'] = true, ensure that prometheus['scrape_configs'] is not set in /etc/gitlab/gitlab.rb. Setting both consul['monitoring_service_discovery'] = true and prometheus['scrape_configs'] in /etc/gitlab/gitlab.rb will result in errors.