gitlab-org--gitlab-foss/doc/operations/incident_management/alert_integrations.md

stage	group	info
Monitor	Health	To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers

Alert integrations

• Introduced in GitLab Ultimate 12.4.
• Moved to GitLab Core in 12.8.

GitLab can accept alerts from any source via a webhook receiver. This can be configured generically or, in GitLab versions 13.1 and greater, you can configure External Prometheus instances to use this endpoint.

Configuration

You can either configure alerts to integrate with an external Prometheus server, or provide a generic HTTP endpoint to receive alerts from other services.

Generic HTTP Endpoint

Enabling the Generic HTTP Endpoint creates a unique HTTP endpoint that can receive alert payloads in JSON format. You can always customize the payload to your liking.

You will need to activate the endpoint and obtain credentials to set up this integration:

1. Sign in to GitLab as a user with maintainer permissions for a project.
2. Navigate to Settings > Operations in your project.
3. Expand the Alerts section, and in the Integration dropdown menu, select Generic.
4. Toggle the Active alert setting to display the URL and Authorization Key for the webhook configuration.

External Prometheus integration

For GitLab versions 13.1 and greater, please see External Prometheus Instances to configure alerts for this integration.

Customizing the payload

You can customize the payload by sending the following parameters. This applies to all types of integrations. All fields other than title are optional:

Property	Type	Description
title	String	The title of the incident. Required.
description	String	A high-level summary of the problem.
start_time	DateTime	The time of the incident. If none is provided, a timestamp of the issue will be used.
end_time	DateTime	For existing alerts only. When provided, the alert is resolved and the associated incident is closed.
service	String	The affected service.
monitoring_tool	String	The name of the associated monitoring tool.
hosts	String or Array	One or more hosts, as to where this incident occurred.
severity	String	The severity of the alert. Must be one of critical, high, medium, low, info, unknown. Default is critical.
fingerprint	String or Array	The unique identifier of the alert. This can be used to group occurrences of the same alert.
gitlab_environment_name	String	The name of the associated GitLab environment. This can be used to associate your alert to your environment.

You can also add custom fields to the alert's payload. The values of extra parameters aren't limited to primitive types (such as strings or numbers), but can be a nested JSON object. For example:

{ "foo": { "bar": { "baz": 42 } } }

TIP: Payload size: Ensure your requests are smaller than the payload application limits.

Example request:

curl --request POST \
  --data '{"title": "Incident title"}' \
  --header "Authorization: Bearer <authorization_key>" \
  --header "Content-Type: application/json" \
  <url>

The <authorization_key> and <url> values can be found when configuring an alert integration.

Example payload:

{
  "title": "Incident title",
  "description": "Short description of the incident",
  "start_time": "2019-09-12T06:00:55Z",
  "service": "service affected",
  "monitoring_tool": "value",
  "hosts": "value",
  "severity": "high",
  "fingerprint": "d19381d4e8ebca87b55cda6e8eee7385",
  "foo": {
    "bar": {
      "baz": 42
    }
  }
}

Triggering test alerts

Introduced in GitLab Core in 13.2.

After a project maintainer or owner configures an integration, you can trigger a test alert to confirm your integration works properly.

1. Sign in as a user with Developer or greater permissions.
2. Navigate to Settings > Operations in your project.
3. Click Alerts endpoint to expand the section.
4. Enter a sample payload in Alert test payload (valid JSON is required).
5. Click Test alert payload.

GitLab displays an error or success message, depending on the outcome of your test.

Automatic grouping of identical alerts (PREMIUM)

Introduced in GitLab Premium 13.2.

In GitLab versions 13.2 and greater, GitLab groups alerts based on their payload. When an incoming alert contains the same payload as another alert (excluding the start_time and hosts attributes), GitLab groups these alerts together and displays a counter on the Alert Management List and details pages.

If the existing alert is already resolved, GitLab creates a new alert instead.

Link to your Opsgenie Alerts

Introduced in GitLab Premium 13.2.

You can monitor alerts using a GitLab integration with Opsgenie.

If you enable the Opsgenie integration, you can't have other GitLab alert services, such as Generic Alerts or Prometheus alerts, active at the same time.

To enable Opsgenie integration:

1. Sign in as a user with Maintainer or Owner permissions.
2. Navigate to Operations > Alerts.
3. In the Integrations select box, select Opsgenie.
4. Select the Active toggle.
5. In the API URL field, enter the base URL for your Opsgenie integration, such as https://app.opsgenie.com/alert/list.
6. Select Save changes.

After you enable the integration, navigate to the Alerts list page at Operations > Alerts, and then select View alerts in Opsgenie.