gitlab-org--gitlab-foss/config/prometheus/additional_metrics.yml

- group: Response metrics (NGINX Ingress)
  priority: 10
  metrics:
  - title: "Throughput"
    y_label: "Requests / Sec"
    required_metrics:
      - nginx_upstream_responses_total
    weight: 1
    queries:
    - query_range: 'sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code)'
      unit: req / sec
      label: Status Code
      series:
        - label: status_code
          when:
            - value: 2xx
              color: green
            - value: 4xx
              color: orange
            - value: 5xx
              color: red
  - title: "Latency"
    y_label: "Latency (ms)"
    required_metrics:
      - nginx_upstream_response_msecs_avg
    weight: 1
    queries:
    - query_range: 'avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"})'
      label: Average
      unit: ms
  - title: "HTTP Error Rate"
    y_label: "HTTP 500 Errors / Sec"
    required_metrics:
      - nginx_upstream_responses_total
    weight: 1
    queries:
    - query_range: 'sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m]))'
      label: HTTP Errors
      unit: "errors / sec"
- group: Response metrics (HA Proxy)
  priority: 10
  metrics:
  - title: "Throughput"
    y_label: "Requests / Sec"
    required_metrics:
      - haproxy_frontend_http_requests_total
    weight: 1
    queries:
    - query_range: 'sum(rate(haproxy_frontend_http_requests_total{%{environment_filter}}[2m])) by (code)'
      unit: req / sec
      series:
        - label: code
          when:
            - value: 2xx
              color: green
            - value: 4xx
              color: yellow
            - value: 5xx
              color: red
  - title: "HTTP Error Rate"
    y_label: "Error Rate (%)"
    required_metrics:
      - haproxy_frontend_http_responses_total
    weight: 1
    queries:
    - query_range: 'sum(rate(haproxy_frontend_http_responses_total{code="5xx",%{environment_filter}}[2m])) / sum(rate(haproxy_frontend_http_responses_total{%{environment_filter}}[2m]))'
      label: HTTP Errors
      unit: "%"
- group: Response metrics (AWS ELB)
  priority: 10
  metrics:
  - title: "Throughput"
    y_label: "Requests / Sec"
    required_metrics:
      - aws_elb_request_count_sum
    weight: 1
    queries:
    - query_range: 'sum(aws_elb_request_count_sum{%{environment_filter}}) / 60'
      label: Total
      unit: req / sec
  - title: "Latency"
    y_label: "Latency (ms)"
    required_metrics:
      - aws_elb_latency_average
    weight: 1
    queries:
    - query_range: 'avg(aws_elb_latency_average{%{environment_filter}}) * 1000'
      label: Average
      unit: ms
  - title: "HTTP Error Rate"
    y_label: "Error Rate (%)"
    required_metrics:
      - aws_elb_request_count_sum
      - aws_elb_httpcode_backend_5_xx_sum
    weight: 1
    queries:
    - query_range: 'sum(aws_elb_httpcode_backend_5_xx_sum{%{environment_filter}}) / sum(aws_elb_request_count_sum{%{environment_filter}})'
      label: HTTP Errors
      unit: "%"
- group: Response metrics (NGINX)
  priority: 10
  metrics:
  - title: "Throughput"
    y_label: "Requests / Sec"
    required_metrics:
      - nginx_responses_total
    weight: 1
    queries:
    - query_range: 'sum(rate(nginx_responses_total{server_zone!="*", server_zone!="_", %{environment_filter}}[2m])) by (status_code)'
      unit: req / sec
      label: Status Code
      series:
        - label: status_code
          when:
            - value: 2xx
              color: green
            - value: 4xx
              color: orange
            - value: 5xx
              color: red
  - title: "Latency"
    y_label: "Latency (ms)"
    required_metrics:
      - nginx_upstream_response_msecs_avg
    weight: 1
    queries:
    - query_range: 'avg(nginx_upstream_response_msecs_avg{%{environment_filter}})'
      label: Upstream
      unit: ms
  - title: "HTTP Error Rate"
    y_label: "HTTP 500 Errors / Sec"
    required_metrics:
      - nginx_responses_total
    weight: 1
    queries:
    - query_range: 'sum(rate(nginx_responses_total{status_code="5xx", %{environment_filter}}[2m]))'
      label: HTTP Errors
      unit: "errors / sec"
- group: System metrics (Kubernetes)
  priority: 5
  metrics:
  - title: "Memory Usage"
    y_label: "Memory Usage (MB)"
    required_metrics:
      - container_memory_usage_bytes
    weight: 1
    queries:
    - query_range: '(sum(avg(container_memory_usage_bytes{container_name!="POD",environment="%{ci_environment_slug}"}) without (job))) / count(avg(container_memory_usage_bytes{container_name!="POD",environment="%{ci_environment_slug}"}) without (job)) /1024/1024'
      label: Average
      unit: MB
  - title: "CPU Utilization"
    y_label: "CPU Utilization (%)"
    required_metrics:
     - container_cpu_usage_seconds_total
    weight: 1
    queries:
    - query_range: 'sum(avg(rate(container_cpu_usage_seconds_total{container_name!="POD",environment="%{ci_environment_slug}"}[2m])) without (job)) * 100'
      label: Average
      unit: "%"
Improve Prometheus docs, include nginx ingress 2017-08-14 12:23:32 -04:00			`- group: Response metrics (NGINX Ingress)`
			`priority: 10`
			`metrics:`
			`- title: "Throughput"`
			`y_label: "Requests / Sec"`
			`required_metrics:`
Update queries for colors and series 2017-09-08 16:16:38 -04:00			`- nginx_upstream_responses_total`
Improve Prometheus docs, include nginx ingress 2017-08-14 12:23:32 -04:00			`weight: 1`
			`queries:`
Fix ingress query 2017-09-09 00:50:02 -04:00			`- query_range: 'sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code)'`
Improve Prometheus docs, include nginx ingress 2017-08-14 12:23:32 -04:00			`unit: req / sec`
Fix ingress query 2017-09-09 00:50:02 -04:00			`label: Status Code`
Update queries for colors and series 2017-09-08 16:16:38 -04:00			`series:`
			`- label: status_code`
			`when:`
			`- value: 2xx`
			`color: green`
			`- value: 4xx`
Fix ingress query 2017-09-09 00:50:02 -04:00			`color: orange`
Update queries for colors and series 2017-09-08 16:16:38 -04:00			`- value: 5xx`
			`color: red`
Improve Prometheus docs, include nginx ingress 2017-08-14 12:23:32 -04:00			`- title: "Latency"`
			`y_label: "Latency (ms)"`
			`required_metrics:`
			`- nginx_upstream_response_msecs_avg`
			`weight: 1`
			`queries:`
			`- query_range: 'avg(nginx_upstream_response_msecs_avg{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"})'`
			`label: Average`
			`unit: ms`
			`- title: "HTTP Error Rate"`
			`y_label: "HTTP 500 Errors / Sec"`
			`required_metrics:`
			`- nginx_upstream_responses_total`
			`weight: 1`
			`queries:`
			`- query_range: 'sum(rate(nginx_upstream_responses_total{status_code="5xx", upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m]))'`
			`label: HTTP Errors`
			`unit: "errors / sec"`
Rename the metric groups 2017-07-27 14:05:58 -04:00			`- group: Response metrics (HA Proxy)`
Add HA Proxy metrics, fix NGINX 2017-07-19 21:22:46 -04:00			`priority: 10`
			`metrics:`
			`- title: "Throughput"`
			`y_label: "Requests / Sec"`
			`required_metrics:`
			`- haproxy_frontend_http_requests_total`
			`weight: 1`
			`queries:`
Update queries for colors and series 2017-09-08 16:16:38 -04:00			`- query_range: 'sum(rate(haproxy_frontend_http_requests_total{%{environment_filter}}[2m])) by (code)'`
Add HA Proxy metrics, fix NGINX 2017-07-19 21:22:46 -04:00			`unit: req / sec`
Update queries for colors and series 2017-09-08 16:16:38 -04:00			`series:`
			`- label: code`
			`when:`
			`- value: 2xx`
			`color: green`
			`- value: 4xx`
			`color: yellow`
			`- value: 5xx`
			`color: red`
Add HA Proxy metrics, fix NGINX 2017-07-19 21:22:46 -04:00			`- title: "HTTP Error Rate"`
			`y_label: "Error Rate (%)"`
			`required_metrics:`
			`- haproxy_frontend_http_responses_total`
			`weight: 1`
			`queries:`
Fix for HA Proxy 2017-07-20 01:51:31 -04:00			`- query_range: 'sum(rate(haproxy_frontend_http_responses_total{code="5xx",%{environment_filter}}[2m])) / sum(rate(haproxy_frontend_http_responses_total{%{environment_filter}}[2m]))'`
Add HA Proxy metrics, fix NGINX 2017-07-19 21:22:46 -04:00			`label: HTTP Errors`
			`unit: "%"`
Rename the metric groups 2017-07-27 14:05:58 -04:00			`- group: Response metrics (AWS ELB)`
Add ELB metrics 2017-07-03 16:28:08 -04:00			`priority: 10`
			`metrics:`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`- title: "Throughput"`
			`y_label: "Requests / Sec"`
Add ELB metrics 2017-07-03 16:28:08 -04:00			`required_metrics:`
			`- aws_elb_request_count_sum`
			`weight: 1`
			`queries:`
Fix incorrect AWS ELB metrics. 2017-07-16 20:52:37 -04:00			`- query_range: 'sum(aws_elb_request_count_sum{%{environment_filter}}) / 60'`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`label: Total`
			`unit: req / sec`
Add ELB metrics 2017-07-03 16:28:08 -04:00			`- title: "Latency"`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`y_label: "Latency (ms)"`
Add ELB metrics 2017-07-03 16:28:08 -04:00			`required_metrics:`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`- aws_elb_latency_average`
Add ELB metrics 2017-07-03 16:28:08 -04:00			`weight: 1`
			`queries:`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`- query_range: 'avg(aws_elb_latency_average{%{environment_filter}}) * 1000'`
			`label: Average`
			`unit: ms`
			`- title: "HTTP Error Rate"`
			`y_label: "Error Rate (%)"`
Add ELB metrics 2017-07-03 16:28:08 -04:00			`required_metrics:`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`- aws_elb_request_count_sum`
			`- aws_elb_httpcode_backend_5_xx_sum`
Add ELB metrics 2017-07-03 16:28:08 -04:00			`weight: 1`
			`queries:`
			`- query_range: 'sum(aws_elb_httpcode_backend_5_xx_sum{%{environment_filter}}) / sum(aws_elb_request_count_sum{%{environment_filter}})'`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`label: HTTP Errors`
			`unit: "%"`
Rename the metric groups 2017-07-27 14:05:58 -04:00			`- group: Response metrics (NGINX)`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`priority: 10`
Additional metrics initial work, with working metrics listing, but without actoual metrics mesurements 2017-05-10 05:25:30 -04:00			`metrics:`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`- title: "Throughput"`
			`y_label: "Requests / Sec"`
Matched Metrics tests 2017-05-31 13:36:07 -04:00			`required_metrics:`
Update queries for colors and series 2017-09-08 16:16:38 -04:00			`- nginx_responses_total`
Additional metrics initial work, with working metrics listing, but without actoual metrics mesurements 2017-05-10 05:25:30 -04:00			`weight: 1`
			`queries:`
Update queries for colors and series 2017-09-08 16:16:38 -04:00			`- query_range: 'sum(rate(nginx_responses_total{server_zone!="*", server_zone!="_", %{environment_filter}}[2m])) by (status_code)'`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`unit: req / sec`
Update queries for colors and series 2017-09-08 16:16:38 -04:00			`label: Status Code`
			`series:`
			`- label: status_code`
			`when:`
			`- value: 2xx`
			`color: green`
			`- value: 4xx`
			`color: orange`
			`- value: 5xx`
			`color: red`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`- title: "Latency"`
			`y_label: "Latency (ms)"`
Matched Metrics tests 2017-05-31 13:36:07 -04:00			`required_metrics:`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`- nginx_upstream_response_msecs_avg`
Additional metrics initial work, with working metrics listing, but without actoual metrics mesurements 2017-05-10 05:25:30 -04:00			`weight: 1`
			`queries:`
Improve Prometheus docs, include nginx ingress 2017-08-14 12:23:32 -04:00			`- query_range: 'avg(nginx_upstream_response_msecs_avg{%{environment_filter}})'`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`label: Upstream`
			`unit: ms`
			`- title: "HTTP Error Rate"`
Improve Prometheus docs, include nginx ingress 2017-08-14 12:23:32 -04:00			`y_label: "HTTP 500 Errors / Sec"`
Matched Metrics tests 2017-05-31 13:36:07 -04:00			`required_metrics:`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`- nginx_responses_total`
			`weight: 1`
			`queries:`
Improve Prometheus docs, include nginx ingress 2017-08-14 12:23:32 -04:00			`- query_range: 'sum(rate(nginx_responses_total{status_code="5xx", %{environment_filter}}[2m]))'`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`label: HTTP Errors`
Improve Prometheus docs, include nginx ingress 2017-08-14 12:23:32 -04:00			`unit: "errors / sec"`
Rename the metric groups 2017-07-27 14:05:58 -04:00			`- group: System metrics (Kubernetes)`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`priority: 5`
			`metrics:`
			`- title: "Memory Usage"`
			`y_label: "Memory Usage (MB)"`
			`required_metrics:`
			`- container_memory_usage_bytes`
Additional metrics initial work, with working metrics listing, but without actoual metrics mesurements 2017-05-10 05:25:30 -04:00			`weight: 1`
			`queries:`
Fix to use variable 2017-11-07 12:29:39 -05:00			`- query_range: '(sum(avg(container_memory_usage_bytes{container_name!="POD",environment="%{ci_environment_slug}"}) without (job))) / count(avg(container_memory_usage_bytes{container_name!="POD",environment="%{ci_environment_slug}"}) without (job)) /1024/1024'`
Add NGINX metrics and other minor changes 2017-07-04 01:45:35 -04:00			`label: Average`
			`unit: MB`
			`- title: "CPU Utilization"`
			`y_label: "CPU Utilization (%)"`
Matched Metrics tests 2017-05-31 13:36:07 -04:00			`required_metrics:`
			`- container_cpu_usage_seconds_total`
Additional metrics initial work, with working metrics listing, but without actoual metrics mesurements 2017-05-10 05:25:30 -04:00			`weight: 1`
			`queries:`
Fix CPU metric 2017-11-07 12:34:34 -05:00			`- query_range: 'sum(avg(rate(container_cpu_usage_seconds_total{container_name!="POD",environment="%{ci_environment_slug}"}[2m])) without (job)) * 100'`
Change label to be more consistent, not a series. 2017-11-12 00:19:43 -05:00			`label: Average`
			`unit: "%"`