gitlab-org--gitlab-foss/doc/api
Yorick Peterse 0395c47193
Migrate events into a new format
This commit migrates events data in such a way that push events are
stored much more efficiently. This is done by creating a shadow table
called "events_for_migration", and a table called "push_event_payloads"
which is used for storing push data of push events. The background
migration in this commit will copy events from the "events" table into
the "events_for_migration" table, push events in will also have a row
created in "push_event_payloads".

This approach allows us to reclaim space in the next release by simply
swapping the "events" and "events_for_migration" tables, then dropping
the old events (now "events_for_migration") table.

The new table structure is also optimised for storage space, and does
not include the unused "title" column nor the "data" column (since this
data is moved to "push_event_payloads").

== Newly Created Events

Newly created events are inserted into both "events" and
"events_for_migration", both using the exact same primary key value. The
table "push_event_payloads" in turn has a foreign key to the _shadow_
table. This removes the need for recreating and validating the foreign
key after swapping the tables. Since the shadow table also has a foreign
key to "projects.id" we also don't have to worry about orphaned rows.

This approach however does require some additional storage as we're
duplicating a portion of the events data for at least 1 release. The
exact amount is hard to estimate, but for GitLab.com this is expected to
be between 10 and 20 GB at most. The background migration in this commit
deliberately does _not_ update the "events" table as doing so would put
a lot of pressure on PostgreSQL's auto vacuuming system.

== Supporting Both Old And New Events

Application code has also been adjusted to support push events using
both the old and new data formats. This is done by creating a PushEvent
class which extends the regular Event class. Using Rails' Single Table
Inheritance system we can ensure the right class is used for the right
data, which in this case is based on the value of `events.action`. To
support displaying old and new data at the same time the PushEvent class
re-defines a few methods of the Event class, falling back to their
original implementations for push events in the old format.

Once all existing events have been migrated the various push event
related methods can be removed from the Event model, and the calls to
`super` can be removed from the methods in the PushEvent model.

The UI and event atom feed have also been slightly changed to better
handle this new setup, fortunately only a few changes were necessary to
make this work.

== API Changes

The API only displays push data of events in the new format. Supporting
both formats in the API is a bit more difficult compared to the UI.
Since the old push data was not really well documented (apart from one
example that used an incorrect "action" nmae) I decided that supporting
both was not worth the effort, especially since events will be migrated
in a few days _and_ new events are created in the correct format.
2017-08-10 17:45:44 +02:00
..
ci Add link to doc/api/ci/lint.md 2017-07-17 22:32:24 +09:00
templates Add "API" to all respective headings (h1s) 2017-05-18 08:49:03 +00:00
access_requests.md update API doc with +API (h1 heading) 2017-05-10 15:48:07 -03:00
award_emoji.md Add "API" to all respective headings (h1s) 2017-05-18 08:49:03 +00:00
boards.md Add "API" to all respective headings (h1s) 2017-05-18 08:49:03 +00:00
branches.md Extending API for protected branches 2017-08-02 10:16:17 +00:00
broadcast_messages.md Add "API" to all respective headings (h1s) 2017-05-18 08:49:03 +00:00
build_triggers.md
builds.md
commits.md Add start_branch to files and commits APIs 2017-08-04 19:18:07 +02:00
deploy_key_multiple_projects.md Add "API" to all respective headings (h1s) 2017-05-18 08:49:03 +00:00
deploy_keys.md Add "API" to all respective headings (h1s) 2017-05-18 08:49:03 +00:00
deployments.md Update documentation to reflect the removal of is_admin from most API responses. 2017-04-25 09:46:05 +00:00
environments.md Resolve "API: Environment info missed" 2017-06-06 10:23:29 +00:00
events.md Migrate events into a new format 2017-08-10 17:45:44 +02:00
features.md Improve Features API and its docs and add a Changelog item 2017-07-06 11:18:26 +02:00
group_level_variables.md Document update 2017-07-19 20:57:27 +09:00
group_milestones.md Fix the iids parameters in the API documentation 2017-08-07 22:43:41 +02:00
groups.md update API doc with +API (h1 heading) 2017-05-10 15:48:07 -03:00
issues.md Fix the iids parameters in the API documentation 2017-08-07 22:43:41 +02:00
jobs.md #32340 - Include mention of "manual" scope in jobs api doc 2017-05-16 15:17:30 +00:00
keys.md update API doc with +API (h1 heading) 2017-05-10 15:48:07 -03:00
labels.md update API doc with +API (h1 heading) 2017-05-10 15:48:07 -03:00
members.md update API doc with +API (h1 heading) 2017-05-10 15:48:07 -03:00
merge_requests.md Fix the iids parameters in the API documentation 2017-08-07 22:43:41 +02:00
milestones.md Fix the iids parameters in the API documentation 2017-08-07 22:43:41 +02:00
namespaces.md Improve tests text 2017-06-28 17:55:50 -03:00
notes.md Expose noteable_iid in Note 2017-08-08 11:31:55 +00:00
notification_settings.md update API doc with +API (h1 heading) 2017-05-10 15:48:07 -03:00
oauth2.md #20628 Enable implicit flow in Gitlab as OAuth Provider 2017-07-07 16:54:48 +02:00
pipeline_schedules.md Fix document according to the new change 2017-05-30 23:55:09 +09:00
pipeline_triggers.md Add "API" to all respective headings (h1s) 2017-05-18 08:49:03 +00:00
pipelines.md Improve documentation 2017-05-03 02:11:51 +09:00
project_level_variables.md Document update 2017-07-19 20:57:27 +09:00
project_snippets.md add spec and project snippet user agent details endpoint 2017-07-07 15:54:39 +02:00
projects.md Resolve "More RESTful API: include resource URLs in responses" 2017-07-25 09:35:45 +00:00
protected_branches.md Extending API for protected branches 2017-08-02 10:16:17 +00:00
README.md Merge branch 'patch-1' into 'master' 2017-08-03 14:08:22 +00:00
repositories.md Add "API" to all respective headings (h1s) 2017-05-18 08:49:03 +00:00
repository_files.md Add start_branch to files and commits APIs 2017-08-04 19:18:07 +02:00
repository_storage_health.md Add a Circuitbreaker for storage paths 2017-08-04 15:38:48 +02:00
runners.md Fix wrong copy edits for URL-encoded substitutions in API docs 2017-04-10 11:17:04 +02:00
services.md Add "API" to all respective headings (h1s) 2017-05-18 08:49:03 +00:00
session.md Add docs for personal access tokens 2017-06-14 08:40:21 +02:00
settings.md Remove deprecated repository_storage attribute 2017-07-27 11:49:27 +02:00
sidekiq_metrics.md Add "API" to all respective headings (h1s) 2017-05-18 08:49:03 +00:00
snippets.md add spec and project snippet user agent details endpoint 2017-07-07 15:54:39 +02:00
system_hooks.md Add "API" to all respective headings (h1s) 2017-05-18 08:49:03 +00:00
tags.md Fix the /projects/:id/repository/tags endpoint to handle dots in the tag name when the project full path contains a / 2017-08-08 11:50:58 +02:00
todos.md update API doc with +API (h1 heading) 2017-05-10 15:48:07 -03:00
users.md Update documentation of user creation by replacing the 'confirm' param with 'skip_confirmation' 2017-07-28 09:53:09 -06:00
v3_to_v4.md v3 API is unsupported after 9.5, but may not be removed 2017-07-25 11:50:09 +01:00
version.md

GitLab API

Automate GitLab via a simple and powerful API. All definitions can be found under /lib/api.

Resources

Documentation for various API resources can be found separately in the following locations:

The following documentation is for the internal CI API:

Road to GraphQL

Going forward, we will start on moving to GraphQL and deprecate the use of controller-specific endpoints. GraphQL has a number of benefits:

  1. We avoid having to maintain two different APIs.
  2. Callers of the API can request only what they need.
  3. It is versioned by default.

It will co-exist with the current v4 REST API. If we have a v5 API, this should be a compatibility layer on top of GraphQL.

Basic usage

API requests should be prefixed with api and the API version. The API version is defined in lib/api.rb. For example, the root of the v4 API is at /api/v4.

For endpoints that require authentication, you need to pass a private_token parameter via query string or header. If passed as a header, the header name must be PRIVATE-TOKEN (uppercase and with a dash instead of an underscore).

Example of a valid API request:

GET /projects?private_token=9koXpg98eAheJpvBs5tK

Example of a valid API request using cURL and authentication via header:

curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects"

Example of a valid API request using cURL and authentication via a query string:

curl "https://gitlab.example.com/api/v4/projects?private_token=9koXpg98eAheJpvBs5tK"

The API uses JSON to serialize data. You don't need to specify .json at the end of an API URL.

Authentication

Most API requests require authentication via a session cookie or token. For those cases where it is not required, this will be mentioned in the documentation for each individual endpoint. For example, the /projects/:id endpoint.

There are three types of access tokens available:

  1. OAuth2 tokens
  2. Private tokens
  3. Personal access tokens

If authentication information is invalid or omitted, an error message will be returned with status code 401:

{
  "message": "401 Unauthorized"
}

When signing in to GitLab as an ordinary user, a _gitlab_session cookie is set. The API will use this cookie for authentication if it is present, but using the API to generate a new session cookie is currently not supported.

OAuth2 tokens

You can use an OAuth 2 token to authenticate with the API by passing it either in the access_token parameter or in the Authorization header.

Example of using the OAuth2 token in the header:

curl --header "Authorization: Bearer OAUTH-TOKEN" https://gitlab.example.com/api/v4/projects

Read more about GitLab as an OAuth2 client.

Private tokens

Private tokens provide full access to the GitLab API. Anyone with access to them can interact with GitLab as if they were you. You can find or reset your private token in your account page (/profile/account).

For examples of usage, read the basic usage section.

Personal access tokens

Instead of using your private token which grants full access to your account, personal access tokens could be a better fit because of their granular permissions.

Once you have your token, pass it to the API using either the private_token parameter or the PRIVATE-TOKEN header. For examples of usage, read the basic usage section.

Read more about personal access tokens.

Impersonation tokens

Introduced in GitLab 9.0. Needs admin permissions.

Impersonation tokens are a type of personal access token that can only be created by an admin for a specific user.

They are a better alternative to using the user's password/private token or using the Sudo feature which also requires the admin's password or private token, since the password/token can change over time. Impersonation tokens are a great fit if you want to build applications or tools which authenticate with the API as a specific user.

For more information, refer to the users API docs.

For examples of usage, read the basic usage section.

Sudo

Needs admin permissions.

All API requests support performing an API call as if you were another user, provided your private token is from an administrator account. You need to pass the sudo parameter either via query string or a header with an ID/username of the user you want to perform the operation as. If passed as a header, the header name must be SUDO (uppercase).

If a non administrative private_token is provided, then an error message will be returned with status code 403:

{
  "message": "403 Forbidden - Must be admin to use sudo"
}

If the sudo user ID or username cannot be found, an error message will be returned with status code 404:

{
  "message": "404 Not Found: No user id or username for: <id/username>"
}

Example of a valid API call and a request using cURL with sudo request, providing a username:

GET /projects?private_token=9koXpg98eAheJpvBs5tK&sudo=username
curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --header "SUDO: username" "https://gitlab.example.com/api/v4/projects"

Example of a valid API call and a request using cURL with sudo request, providing an ID:

GET /projects?private_token=9koXpg98eAheJpvBs5tK&sudo=23
curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --header "SUDO: 23" "https://gitlab.example.com/api/v4/projects"

Status codes

The API is designed to return different status codes according to context and action. This way, if a request results in an error, the caller is able to get insight into what went wrong.

The following table gives an overview of how the API functions generally behave.

Request type Description
GET Access one or more resources and return the result as JSON.
POST Return 201 Created if the resource is successfully created and return the newly created resource as JSON.
GET / PUT / DELETE Return 200 OK if the resource is accessed, modified or deleted successfully. The (modified) result is returned as JSON.
DELETE Designed to be idempotent, meaning a request to a resource still returns 200 OK even it was deleted before or is not available. The reasoning behind this, is that the user is not really interested if the resource existed before or not.

The following table shows the possible return codes for API requests.

Return values Description
200 OK The GET, PUT or DELETE request was successful, the resource(s) itself is returned as JSON.
204 No Content The server has successfully fulfilled the request and that there is no additional content to send in the response payload body.
201 Created The POST request was successful and the resource is returned as JSON.
304 Not Modified Indicates that the resource has not been modified since the last request.
400 Bad Request A required attribute of the API request is missing, e.g., the title of an issue is not given.
401 Unauthorized The user is not authenticated, a valid user token is necessary.
403 Forbidden The request is not allowed, e.g., the user is not allowed to delete a project.
404 Not Found A resource could not be accessed, e.g., an ID for a resource could not be found.
405 Method Not Allowed The request is not supported.
409 Conflict A conflicting resource already exists, e.g., creating a project with a name that already exists.
422 Unprocessable The entity could not be processed.
500 Server Error While handling the request something went wrong server-side.

Pagination

Sometimes the returned result will span across many pages. When listing resources you can pass the following parameters:

Parameter Description
page Page number (default: 1)
per_page Number of items to list per page (default: 20, max: 100)

In the example below, we list 50 namespaces per page.

curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/namespaces?per_page=50

Link headers are sent back with each response. They have rel set to prev/next/first/last and contain the relevant URL. Please use these links instead of generating your own URLs.

In the cURL example below, we limit the output to 3 items per page (per_page=3) and we request the second page (page=2) of comments of the issue with ID 8 which belongs to the project with ID 8:

curl --head --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/8/issues/8/notes?per_page=3&page=2

The response will then be:

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 1103
Content-Type: application/json
Date: Mon, 18 Jan 2016 09:43:18 GMT
Link: <https://gitlab.example.com/api/v4/projects/8/issues/8/notes?page=1&per_page=3>; rel="prev", <https://gitlab.example.com/api/v4/projects/8/issues/8/notes?page=3&per_page=3>; rel="next", <https://gitlab.example.com/api/v4/projects/8/issues/8/notes?page=1&per_page=3>; rel="first", <https://gitlab.example.com/api/v4/projects/8/issues/8/notes?page=3&per_page=3>; rel="last"
Status: 200 OK
Vary: Origin
X-Next-Page: 3
X-Page: 2
X-Per-Page: 3
X-Prev-Page: 1
X-Request-Id: 732ad4ee-9870-4866-a199-a9db0cde3c86
X-Runtime: 0.108688
X-Total: 8
X-Total-Pages: 3

Other pagination headers

Additional pagination headers are also sent back.

Header Description
X-Total The total number of items
X-Total-Pages The total number of pages
X-Per-Page The number of items per page
X-Page The index of the current page (starting at 1)
X-Next-Page The index of the next page
X-Prev-Page The index of the previous page

Namespaced path encoding

If using namespaced API calls, make sure that the NAMESPACE/PROJECT_NAME is URL-encoded.

For example, / is represented by %2F:

GET /api/v4/projects/diaspora%2Fdiaspora

Branches & tags name encoding

If your branch or tag contains a /, make sure the branch/tag name is URL-encoded.

For example, / is represented by %2F:

GET /api/v4/projects/1/branches/my%2Fbranch/commits

id vs iid

When you work with the API, you may notice two similar fields in API entities: id and iid. The main difference between them is scope.

For example, an issue might have id: 46 and iid: 5.

Parameter Description
id Is unique across all issues and is used for any API call
iid Is unique only in scope of a single project. When you browse issues or merge requests with the Web UI, you see the iid

That means that if you want to get an issue via the API you should use the id:

GET /projects/42/issues/:id

On the other hand, if you want to create a link to a web page you should use the iid:

GET /projects/42/issues/:iid

Data validation and error reporting

When working with the API you may encounter validation errors, in which case the API will answer with an HTTP 400 status.

Such errors appear in two cases:

  • A required attribute of the API request is missing, e.g., the title of an issue is not given
  • An attribute did not pass the validation, e.g., user bio is too long

When an attribute is missing, you will get something like:

HTTP/1.1 400 Bad Request
Content-Type: application/json
{
    "message":"400 (Bad request) \"title\" not given"
}

When a validation error occurs, error messages will be different. They will hold all details of validation errors:

HTTP/1.1 400 Bad Request
Content-Type: application/json
{
    "message": {
        "bio": [
            "is too long (maximum is 255 characters)"
        ]
    }
}

This makes error messages more machine-readable. The format can be described as follows:

{
    "message": {
        "<property-name>": [
            "<error-message>",
            "<error-message>",
            ...
        ],
        "<embed-entity>": {
            "<property-name>": [
                "<error-message>",
                "<error-message>",
                ...
            ],
        }
    }
}

Unknown route

When you try to access an API URL that does not exist you will receive 404 Not Found.

HTTP/1.1 404 Not Found
Content-Type: application/json
{
    "error": "404 Not Found"
}

Clients

There are many unofficial GitLab API Clients for most of the popular programming languages. Visit the GitLab website for a complete list.