Update Prometheus docs for enhanced integration
This commit is contained in:
Joshua Lambert
parent
b44b4d4d64
commit
a55106d629
5 changed files with 52 additions and 206 deletions
BIN
doc/user/project/integrations/img/prometheus_dashboard.png
Normal file
BIN
doc/user/project/integrations/img/prometheus_dashboard.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 62 KiB |
BIN
doc/user/project/integrations/img/prometheus_deploy.png
Normal file
BIN
doc/user/project/integrations/img/prometheus_deploy.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
|
|
@ -2,17 +2,54 @@
|
|||
|
||||
> [Introduced][ce-8935] in GitLab 9.0.
|
||||
|
||||
GitLab offers powerful integration with [Prometheus] for monitoring your apps.
|
||||
Metrics are retrieved from the configured Prometheus server, and then displayed
|
||||
GitLab offers powerful integration with [Prometheus] for monitoring key metrics your apps, directly within GitLab.
|
||||
Metrics for each environment are retrieved from Prometheus, and then displayed
|
||||
within the GitLab interface.
|
||||
|
||||
Each project can be configured with its own specific Prometheus server, see the
|
||||
[configuration](#configuration) section for more details. If you have a single
|
||||
Prometheus server which monitors all of your infrastructure, you can pre-fill
|
||||
the settings page with a default template. To configure the template, see the
|
||||
[Services templates](services_templates.md) document.
|
||||
|
||||
|
||||
## Requirements
|
||||
There are two ways to setup Prometheus integration, depending on where your apps are running:
|
||||
* For deployments on Kubernetes, GitLab can [deploy and manage Prometheus](#managed-prometheus-on-kubernetes) in a cluster
|
||||
* For other deployment targets, simply [specify the Prometheus server](manual-configuration-of-prometheus).
|
||||
|
||||
## Managed Prometheus on Kubernetes
|
||||
|
||||
GitLab can seamlessly deploy and manage Prometheus on a [connected Kubernetes cluster](../clusters/index.html), making monitoring the metrics of your deployed apps as easy as a single click.
|
||||
|
||||
### Requirements
|
||||
|
||||
* GitLab [10.5 or above](https://gitlab.com/gitlab-org/gitlab-ce/issues/28916)
|
||||
* A [connected Kubernetes cluster](../clusters/index.html)
|
||||
* Helm Tiller [installed by GitLab](../clusters/index.html#installing-applications)
|
||||
|
||||
### Getting started
|
||||
|
||||
Once you have a connected Kubernetes cluster with Helm installed, deploying a managed Prometheus is as easy as a single click.
|
||||
|
||||
1. Go to the `CI/CD > Kubernetes` page, to view your connected clusters
|
||||
1. Select the cluster you would like to deploy Prometheus to
|
||||
1. Click the **Install** button to deploy Prometheus to the cluster
|
||||
|
||||
|
||||
|
||||
### About managed Prometheus deployments
|
||||
|
||||
Prometheus is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/prometheus). Prometheus is only accessible within the cluster, with GitLab communicating through the [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/).
|
||||
|
||||
The Prometheus server will [automatically detect and monitor](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Ckubernetes_sd_config%3E) nodes, pods, and endpoints.
|
||||
|
||||
To configure a resource to be monitored by Prometheus, simply set the following [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/):
|
||||
* `prometheus.io/scrape` to `true` to enable monitoring of the resource.
|
||||
* `prometheus.io/port` to define the port of the metrics endpoint.
|
||||
* `prometheus.io/path` to define the path of the metrics endpoint. Defaults to `/metrics`.
|
||||
|
||||
CPU and Memory consumption is also monitored, but requires [naming conventions](prometheus_library/kubernetes.html#specifying-the-environment) in order to determine the environment. If you are using [Auto DevOps](../../../topics/autodevops/), this is handled automatically.
|
||||
|
||||
The [NGINX Ingress]((../clusters/index.html#installing-applications)) that is deployed by GitLab to clusters, is automatically annotated for monitoring providing key response metrics: latency, throughput, and error rates.
|
||||
|
||||
## Manual configuration of Prometheus
|
||||
|
||||
### Requirements
|
||||
|
||||
Integration with Prometheus requires the following:
|
||||
|
||||
|
|
@ -21,15 +58,7 @@ Integration with Prometheus requires the following:
|
|||
1. Each metric must be have a label to indicate the environment
|
||||
1. GitLab must have network connectivity to the Prometheus server
|
||||
|
||||
## Getting started with Prometheus monitoring
|
||||
|
||||
Depending on your deployment and where you have located your GitLab server, there are a few options to get started with Prometheus monitoring.
|
||||
|
||||
* If both GitLab and your applications are installed in the same Kubernetes cluster, you can leverage the [bundled Prometheus server within GitLab](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes).
|
||||
* If your applications are deployed on Kubernetes, but GitLab is not in the same cluster, then you can [configure a Prometheus server in your Kubernetes cluster](#configuring-your-own-prometheus-server-within-kubernetes).
|
||||
* If your applications are not running in Kubernetes, [get started with Prometheus](#getting-started-with-prometheus-outside-of-kubernetes).
|
||||
|
||||
### Getting started with Prometheus outside of Kubernetes
|
||||
### Getting started
|
||||
|
||||
Installing and configuring Prometheus to monitor applications is fairly straight forward.
|
||||
|
||||
|
|
@ -37,84 +66,7 @@ Installing and configuring Prometheus to monitor applications is fairly straight
|
|||
1. Set up one of the [supported monitoring targets](prometheus_library/metrics.md)
|
||||
1. Configure the Prometheus server to [collect their metrics](https://prometheus.io/docs/operating/configuration/#scrape_config)
|
||||
|
||||
### Configuring Omnibus GitLab Prometheus to monitor Kubernetes deployments
|
||||
|
||||
With Omnibus GitLab running inside of Kubernetes, you can leverage the bundled
|
||||
version of Prometheus to collect the supported metrics. Once enabled, Prometheus will automatically begin monitoring Kubernetes Nodes and any [annotated Pods](https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>).
|
||||
|
||||
1. Read how to configure the bundled Prometheus server in the
|
||||
[Administration guide][gitlab-prometheus-k8s-monitor].
|
||||
1. Now that Prometheus is configured, proceed on
|
||||
[configuring the Prometheus project service in GitLab](#configuration-in-gitlab).
|
||||
|
||||
### Configuring your own Prometheus server within Kubernetes
|
||||
|
||||
Setting up and configuring Prometheus within Kubernetes is quick and painless.
|
||||
The Prometheus project provides an [official Docker image][prometheus-docker-image]
|
||||
which we can use as a starting point.
|
||||
|
||||
To get started quickly, we have provided a [sample YML file][prometheus-yml]
|
||||
that can be used as a template. This file will create a `prometheus` **Namespace**,
|
||||
**Service**, **Deployment**, and **ConfigMap** in Kubernetes. You can upload
|
||||
this file to the Kubernetes dashboard using **+ Create** at the top right.
|
||||
|
||||
|
||||
|
||||
Or use `kubectl`:
|
||||
|
||||
```bash
|
||||
kubectl apply -f path/to/prometheus.yml
|
||||
```
|
||||
|
||||
Once deployed, you should see the Prometheus service, deployment, and
|
||||
pod start within the `prometheus` namespace. The server will begin to collect
|
||||
metrics from each Kubernetes Node in the cluster, based on the configuration
|
||||
provided in the template. It will also attempt to collect metrics from any Kubernetes Pods that have been [annotated for Prometheus](https://prometheus.io/docs/operating/configuration/#pod).
|
||||
|
||||
Since GitLab is not running within Kubernetes, the template provides external
|
||||
network access via a `NodePort` running on `30090`. This method allows access
|
||||
to be controlled using provider firewall rules, like within Google Compute Engine.
|
||||
|
||||
Since a `NodePort` does not automatically have firewall rules created for it,
|
||||
one will need to be created manually to allow access. In GCP/GKE, you will want
|
||||
to confirm the Node that the Prometheus pod is running on. This can be done
|
||||
either by looking at the Pod in the Kubernetes dashboard, or by running:
|
||||
|
||||
```bash
|
||||
kubectl describe pods -n prometheus
|
||||
```
|
||||
|
||||
Next on GKE, we need to get the `tag` of the Node or VM Instance, so we can
|
||||
create an accurate firewall rule. The easiest way to do this is to go into the
|
||||
Google Cloud Platform Compute console and select the VM instance that matches
|
||||
the name of the Node gathered from the step above. In this case, the node tag
|
||||
needed is `gke-prometheus-demo-5d5ada10-node`. Also make a note of the
|
||||
**External IP**, which will be the IP address the Prometheus server is reachable
|
||||
on.
|
||||
|
||||
|
||||
|
||||
Armed with the proper Node tag, the firewall rule can now be created
|
||||
specifically for this node. To create the firewall rule, open the Google Cloud
|
||||
Platform Networking console, and select **Firewall Rules**.
|
||||
|
||||
Create a new rule:
|
||||
|
||||
- Specify the source IP range to match your desired access list, which should
|
||||
include your GitLab server. A sample of GitLab.com's IP address range is
|
||||
available [in this issue][gitlab.com-ip-range], but note that GitLab.com's IPs
|
||||
are subject to change without prior notification.
|
||||
- Allowed protocol and port should be `tcp:30090`.
|
||||
- The target tags should match the Node tag identified earlier in this step.
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
Now that Prometheus is configured, proceed to
|
||||
[configure the Prometheus project service in GitLab](##configuration-in-gitlab).
|
||||
|
||||
## Configuration in GitLab
|
||||
### Configuration in GitLab
|
||||
|
||||
The actual configuration of Prometheus integration within GitLab is very simple.
|
||||
All you will need is the DNS or IP address of the Prometheus server you'd like
|
||||
|
|
|
|||
|
|
@ -24,9 +24,10 @@ Prometheus server up and running. You have two options here:
|
|||
- If you have an Omnibus based GitLab installation within your Kubernetes cluster, you can leverage the bundled Prometheus server to [monitor Kubernetes](../../../../administration/monitoring/prometheus/index.md#configuring-prometheus-to-monitor-kubernetes).
|
||||
- To configure your own Prometheus server, you can follow the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/) or [our guide](../../../../administration/monitoring/prometheus/index.md#configuring-your-own-prometheus-server-within-kubernetes).
|
||||
|
||||
## Specifying the Environment label
|
||||
## Specifying the Environment
|
||||
|
||||
In order to isolate and only display relevant metrics for a given environment
|
||||
however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments).
|
||||
In order to isolate and only display relevant CPU and Memory metrics for a given environment, GitLab needs a method to detect which containers it is running. Because these metrics are tracked at the container level, traditional Kubernetes labels are not available.
|
||||
|
||||
If you are using [GitLab Auto-Deploy](../../../../ci/autodeploy/index.md) and one of the two [provided Kubernetes monitoring solutions](../prometheus.md#getting-started-with-prometheus-monitoring), the `environment` label will be automatically added.
|
||||
Instead, the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name should begin with the name of the [enviroment](../../../../environments.html). It can be followed by a `-` and additional content if desired.
|
||||
|
||||
If you are using [GitLab Auto-Deploy](../../../../ci/autodeploy/index.md) and one of the two [provided Kubernetes monitoring solutions](../prometheus.md#getting-started-with-prometheus-monitoring), the naming will be correctly set automatically.
|
||||
|
|
|
|||
|
|
@ -1,107 +0,0 @@
|
|||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: prometheus
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: prometheus
|
||||
data:
|
||||
prometheus.yml: |-
|
||||
scrape_configs:
|
||||
- job_name: 'kubernetes-nodes'
|
||||
scheme: https
|
||||
tls_config:
|
||||
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||||
insecure_skip_verify: true
|
||||
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||||
kubernetes_sd_configs:
|
||||
- role: node
|
||||
metric_relabel_configs:
|
||||
- source_labels: [pod_name]
|
||||
target_label: environment
|
||||
regex: (.+)-.+-.+
|
||||
replacement: $1
|
||||
- job_name: kubernetes-pods
|
||||
tls_config:
|
||||
ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
|
||||
insecure_skip_verify: true
|
||||
bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
api_server: https://kubernetes.default.svc:443
|
||||
tls_config:
|
||||
ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
|
||||
bearer_token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
|
||||
relabel_configs:
|
||||
- source_labels:
|
||||
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
|
||||
action: keep
|
||||
regex: 'true'
|
||||
- source_labels:
|
||||
- __meta_kubernetes_pod_annotation_prometheus_io_path
|
||||
action: replace
|
||||
target_label: __metrics_path__
|
||||
regex: "(.+)"
|
||||
- source_labels:
|
||||
- __address__
|
||||
- __meta_kubernetes_pod_annotation_prometheus_io_port
|
||||
action: replace
|
||||
regex: "([^:]+)(?::[0-9]+)?;([0-9]+)"
|
||||
replacement: "$1:$2"
|
||||
target_label: __address__
|
||||
- action: labelmap
|
||||
regex: __meta_kubernetes_pod_label_(.+)
|
||||
- source_labels:
|
||||
- __meta_kubernetes_namespace
|
||||
action: replace
|
||||
target_label: kubernetes_namespace
|
||||
- source_labels:
|
||||
- __meta_kubernetes_pod_name
|
||||
action: replace
|
||||
target_label: kubernetes_pod_name
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: prometheus
|
||||
spec:
|
||||
selector:
|
||||
app: prometheus
|
||||
ports:
|
||||
- name: prometheus
|
||||
protocol: TCP
|
||||
port: 9090
|
||||
nodePort: 30090
|
||||
type: NodePort
|
||||
---
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: prometheus
|
||||
spec:
|
||||
replicas: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: prometheus
|
||||
spec:
|
||||
containers:
|
||||
- name: prometheus
|
||||
image: prom/prometheus:latest
|
||||
args:
|
||||
- '--config.file=/prometheus-data/prometheus.yml'
|
||||
ports:
|
||||
- name: prometheus
|
||||
containerPort: 9090
|
||||
volumeMounts:
|
||||
- name: data-volume
|
||||
mountPath: /prometheus-data
|
||||
volumes:
|
||||
- name: data-volume
|
||||
configMap:
|
||||
name: prometheus
|
||||
Loading…
Reference in a new issue