4.2 KiB

Raw Blame History

stage	group	info
Platforms	Scalability	To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments

GitLab Application Service Level Indicators (SLIs)

Introduced in GitLab 14.4

It is possible to define Service Level Indicators (SLIs) directly in the Ruby codebase. This keeps the definition of operations and their success close to the implementation and allows the people building features to easily define how these features should be monitored.

Defining an SLI causes 2 Prometheus counters to be emitted from the rails application:

gitlab_sli:<sli name>:total: incremented for each operation.
gitlab_sli:<sli_name>:success_total: incremented for successful operations.

Existing SLIs

rails_request_apdex

Defining a new SLI

An SLI can be defined using the Gitlab::Metrics::Sli class.

Before the first scrape, it is important to have initialized the SLI with all possible label-combinations. This avoid confusing results when using these counters in calculations.

To initialize an SLI, use the .inilialize_sli class method, for example:

Gitlab::Metrics::Sli.initialize_sli(:received_email, [
  {
    feature_category: :issue_tracking,
    email_type: :create_issue
  },
  {
    feature_category: :service_desk,
    email_type: :service_desk
  },
  {
    feature_category: :code_review,
    email_type: :create_merge_request
  }
])

Metrics must be initialized before they get scraped for the first time. This could be done at the start time of the process that will emit them, in which case we need to pay attention not to increase application's boot time too much. This is preferable if possible.

Alternatively, if initializing would take too long, this can be done during the first scrape. We need to make sure we don't do it for every scrape. This can be done as follows:

def initialize_request_slis_if_needed!
  return if Gitlab::Metrics::Sli.initialized?(:rails_request_apdex)
  Gitlab::Metrics::Sli.initialize_sli(:rails_request_apdex, possible_request_labels)
end

Also pay attention to do it for the different metrics endpoints we have. Currently the WebExporter and the HealthController for Rails and SidekiqExporter for Sidekiq.

Tracking operations for an SLI

Tracking an operation in the newly defined SLI can be done like this:

Gitlab::Metrics::Sli[:received_email].increment(
  labels: {
    feature_category: :service_desk,
    email_type: :service_desk
  },
  success: issue_created?
)

Calling #increment on this SLI will increment the total Prometheus counter

gitlab_sli:received_email:total{ feature_category='service_desk', email_type='service_desk' }

If the success: argument passed is truthy, then the success counter will also be incremented:

gitlab_sli:received_email:success_total{ feature_category='service_desk', email_type='service_desk' }

Using the SLI in service monitoring and alerts

When the application is emitting metrics for the new SLI, those need to be consumed in the service catalog to result in alerts, and be included in the error budget for stage groups and GitLab.com's overall availability.

This is currently being worked on in this project. As part of this issue we will update the documentation.

For any question, please don't hesitate to createan issue in the Scalability issue tracker or come find us in #g_scalability on Slack.

4.2 KiB Raw Blame History