2020-07-23 11:09:28 -04:00
---
stage: Package
group: Package
2020-11-26 01:09:20 -05:00
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-07-23 11:09:28 -04:00
---
2020-07-21 14:09:45 -04:00
# Packages
2019-05-05 09:57:21 -04:00
2020-11-20 19:09:36 -05:00
This document guides you through adding another [package management system ](../administration/packages/index.md ) support to GitLab.
2019-05-05 09:57:21 -04:00
2019-09-17 08:06:48 -04:00
See already supported package types in [Packages documentation ](../administration/packages/index.md )
2019-05-05 09:57:21 -04:00
2019-11-14 04:06:15 -05:00
Since GitLab packages' UI is pretty generic, it is possible to add basic new
2019-11-21 07:06:40 -05:00
package system support with solely backend changes. This guide is superficial and does
2019-07-08 23:28:41 -04:00
not cover the way the code should be written. However, you can find a good example
2020-10-27 20:08:34 -04:00
by looking at the following merge requests:
2019-05-05 09:57:21 -04:00
2020-02-06 10:09:11 -05:00
- [NPM registry support ](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/8673 ).
- [Maven repository ](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6607 ).
2020-10-27 20:08:34 -04:00
- [Composer repository for PHP dependencies ](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22415 ).
- [Terraform modules registry ](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18834 ).
- [Instance-level endpoint for Maven repository ](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/8757 ).
2019-05-05 09:57:21 -04:00
## General information
The existing database model requires the following:
2019-07-08 23:28:41 -04:00
- Every package belongs to a project.
2019-05-05 09:57:21 -04:00
- Every package file belongs to a package.
- A package can have one or more package files.
- The package model is based on storing information about the package and its version.
2020-03-09 05:07:45 -04:00
### API endpoints
2019-05-05 09:57:21 -04:00
2020-07-21 14:09:45 -04:00
Package systems work with GitLab via API. For example `lib/api/npm_packages.rb`
2019-07-08 23:28:41 -04:00
implements API endpoints to work with NPM clients. So, the first thing to do is to
2020-07-21 14:09:45 -04:00
add a new `lib/api/your_name_packages.rb` file with API endpoints that are
2019-07-08 23:28:41 -04:00
necessary to make the package system client to work. Usually that means having
endpoints like:
2019-05-05 09:57:21 -04:00
- GET package information.
- GET package file content.
- PUT upload package.
2019-11-14 04:06:15 -05:00
Since the packages belong to a project, it's expected to have project-level endpoint (remote)
2019-07-08 23:28:41 -04:00
for uploading and downloading them. For example:
2019-05-05 09:57:21 -04:00
2020-02-07 22:08:47 -05:00
```plaintext
2019-05-05 09:57:21 -04:00
GET https://gitlab.com/api/v4/projects/< your_project_id > /packages/npm/
PUT https://gitlab.com/api/v4/projects/< your_project_id > /packages/npm/
```
2019-07-08 23:28:41 -04:00
Group-level and instance-level endpoints are good to have but are optional.
2019-05-05 09:57:21 -04:00
2020-03-09 05:07:45 -04:00
#### Remote hierarchy
2019-12-03 16:06:23 -05:00
Packages are scoped within various levels of access, which is generally configured by setting your remote. A
remote endpoint may be set at the project level, meaning when installing packages, only packages belonging to that
2020-11-20 19:09:36 -05:00
project are visible. Alternatively, a group-level endpoint may be used to allow visibility to all packages
2019-12-03 16:06:23 -05:00
within a given group. Lastly, an instance-level endpoint can be used to allow visibility to all packages within an
entire GitLab instance.
2020-11-20 19:09:36 -05:00
Using group and project level endpoints allows for more flexibility in package naming, however, more remotes
have to be managed. Using instance level endpoints requires [stricter naming conventions ](#naming-conventions ).
2019-12-03 16:06:23 -05:00
The current state of existing package registries availability is:
2020-10-12 20:08:42 -04:00
| Repository Type | Project Level | Group Level | Instance Level |
|------------------|---------------|-------------|----------------|
| Maven | Yes | Yes | Yes |
| Conan | Yes | No - [open issue ](https://gitlab.com/gitlab-org/gitlab/-/issues/11679 ) | Yes |
| NPM | No - [open issue ](https://gitlab.com/gitlab-org/gitlab/-/issues/36853 ) | Yes | No - [open issue ](https://gitlab.com/gitlab-org/gitlab/-/issues/36853 ) |
2021-01-27 16:09:12 -05:00
| NuGet | Yes | Yes | No - [open issue ](https://gitlab.com/gitlab-org/gitlab/-/issues/36425 ) |
2020-10-12 20:08:42 -04:00
| PyPI | Yes | No | No |
| Go | Yes | No - [open issue ](https://gitlab.com/gitlab-org/gitlab/-/issues/213900 ) | No - [open-issue ](https://gitlab.com/gitlab-org/gitlab/-/issues/213902 ) |
| Composer | Yes | Yes | No |
| Generic | Yes | No | No |
2019-12-03 16:06:23 -05:00
2020-12-04 16:09:29 -05:00
NOTE:
2020-07-16 02:09:33 -04:00
NPM is currently a hybrid of the instance level and group level.
2019-12-03 16:06:23 -05:00
It is using the top-level group or namespace as the defining portion of the name
(for example, `@my-group-name/my-package-name` ).
2020-12-04 16:09:29 -05:00
NOTE:
2020-07-20 20:09:37 -04:00
Composer package naming scope is Instance Level.
2020-05-29 14:08:26 -04:00
2020-03-09 05:07:45 -04:00
### Naming conventions
2019-11-14 04:06:15 -05:00
2020-11-20 19:09:36 -05:00
To avoid name conflict for instance-level endpoints you must define a package naming convention
2019-11-14 04:06:15 -05:00
that gives a way to identify the project that the package belongs to. This generally involves using the project
2020-05-07 02:09:38 -04:00
ID or full project path in the package name. See
2020-09-28 20:09:59 -04:00
[Conan's naming convention ](../user/packages/conan_repository/index.md#package-recipe-naming-convention-for-instance-remotes ) as an example.
2019-11-14 04:06:15 -05:00
2020-11-20 19:09:36 -05:00
For group and project-level endpoints, naming can be less constrained and it is up to the group and project
2020-03-02 04:07:59 -05:00
members to be certain that there is no conflict between two package names. However, the system should prevent
2019-11-14 04:06:15 -05:00
a user from reusing an existing name within a given scope.
Otherwise, naming should follow the package manager's naming conventions and include a validation in the `package.md`
model for that package type.
2020-03-09 05:07:45 -04:00
### Services and finders
2019-11-14 04:06:15 -05:00
Logic for performing tasks such as creating package or package file records or finding packages should not live
within the API file, but should live in services and finders. Existing services and finders should be used or
extended when possible to keep the common package logic grouped as much as possible.
2019-05-05 09:57:21 -04:00
2020-03-09 05:07:45 -04:00
### Configuration
2019-05-05 09:57:21 -04:00
2019-07-08 23:28:41 -04:00
GitLab has a `packages` section in its configuration file (`gitlab.rb`).
It applies to all package systems supported by GitLab. Usually you don't need
to add anything there.
2019-05-05 09:57:21 -04:00
2019-07-08 23:28:41 -04:00
Packages can be configured to use object storage, therefore your code must support it.
2019-05-05 09:57:21 -04:00
2020-03-09 05:07:45 -04:00
## MVC Approach
The way new package systems are integrated in GitLab is using an [MVC ](https://about.gitlab.com/handbook/values/#minimum-viable-change-mvc ). Therefore, the first iteration should support the bare minimum user actions:
- Authentication
- Uploading a package
- Pulling a package
- Required actions
2020-11-20 19:09:36 -05:00
Required actions are all the additional requests that GitLab needs to handle so the corresponding package manager CLI can work properly. It could be a search feature or an endpoint providing meta information about a package. For example:
2020-03-09 05:07:45 -04:00
- For NuGet, the search request was implemented during the first MVC iteration, to support Visual Studio.
2020-04-30 11:09:46 -04:00
- For NPM, there is a metadata endpoint used by `npm` to get the tarball URL.
2020-03-09 05:07:45 -04:00
For the first MVC iteration, it's recommended to stay at the project level of the [remote hierarchy ](#remote-hierarchy ). Other levels can be tackled with [future Merge Requests ](#future-work ).
There are usually 2 phases for the MVC:
- [Analysis ](#analysis )
- [Implementation ](#implementation )
### Keep iterations small
When implementing a new package manager, it is tempting to create one large merge request containing all of the
necessary endpoints and services necessary to support basic usage. Instead, put the
API endpoints behind a [feature flag ](feature_flags/development.md ) and
submit each endpoint or behavior (download, upload, etc) in a different merge request to shorten the review
process.
### Analysis
During this phase, the idea is to collect as much information as possible about the API used by the package system. Here some aspects that can be useful to include:
- **Authentication**: What authentication mechanisms are available (OAuth, Basic
2020-11-20 19:09:36 -05:00
Authorization, other). Keep in mind that GitLab users often want to use their
2020-03-09 05:07:45 -04:00
[Personal Access Tokens ](../user/profile/personal_access_tokens.md ).
Although not needed for the MVC first iteration, the [CI job tokens ](../user/project/new_ci_build_permissions_model.md#job-token )
have to be supported at some point in the future.
- **Requests**: Which requests are needed to have a working MVC. Ideally, produce
a list of all the requests needed for the MVC (including required actions). Further
investigation could provide an example for each request with the request and the response bodies.
2020-11-20 19:09:36 -05:00
- **Upload**: Carefully analyze how the upload process works. This is likely the most
2020-03-09 05:07:45 -04:00
complex request to implement. A detailed analysis is desired here as uploads can be
encoded in different ways (body or multipart) and can even be in a totally different
format (for example, a JSON structure where the package file is a Base64 value of
a particular field). These different encodings lead to slightly different implementations
on GitLab and GitLab Workhorse. For more detailed information, review [file uploads ](#file-uploads ).
2020-11-20 19:09:36 -05:00
- **Endpoints**: Suggest a list of endpoint URLs to implement in GitLab.
2020-03-09 05:07:45 -04:00
- **Split work**: Suggest a list of changes to do to incrementally build the MVC.
2020-11-20 19:09:36 -05:00
This gives a good idea of how much work there is to be done. Here is an example
2020-03-09 05:07:45 -04:00
list that would need to be adapted on a case by case basis:
1. Empty file structure (API file, base service for this package)
1. Authentication system for "logging in" to the package manager
1. Identify metadata and create applicable tables
1. Workhorse route for [object storage direct upload ](uploads.md#direct-upload )
1. Endpoints required for upload/publish
1. Endpoints required for install/download
1. Endpoints required for required actions
The analysis usually takes a full milestone to complete, though it's not impossible to start the implementation in the same milestone.
2020-05-04 14:10:20 -04:00
In particular, the upload request can have some [requirements in the GitLab Workhorse project ](#file-uploads ). This project has a different release cycle than the rails backend. It's **strongly** recommended that you open an issue there as soon as the upload request analysis is done. This way GitLab Workhorse is already ready when the upload request is implemented on the rails backend.
2020-03-09 05:07:45 -04:00
### Implementation
2020-11-20 19:09:36 -05:00
The implementation of the different Merge Requests varies between different package system integrations. Contributors should take into account some important aspects of the implementation phase.
2020-03-09 05:07:45 -04:00
#### Authentication
The MVC must support [Personal Access Tokens ](../user/profile/personal_access_tokens.md ) right from the start. We currently support two options for these tokens: OAuth and Basic Access.
2020-07-21 14:09:45 -04:00
OAuth authentication is already supported. You can see an example in the [npm API ](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/npm_packages.rb ).
2020-03-09 05:07:45 -04:00
[Basic Access authentication ](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication )
support is done by overriding a specific function in the API helpers, like
2020-07-21 14:09:45 -04:00
[this example in the Conan API ](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/conan_packages.rb ).
2020-03-09 05:07:45 -04:00
For this authentication mechanism, keep in mind that some clients can send an unauthenticated
request first, wait for the 401 Unauthorized response with the [`WWW-Authenticate` ](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/WWW-Authenticate )
field, then send an updated (authenticated) request. This case is more involved as
2020-07-21 14:09:45 -04:00
GitLab needs to handle the 401 Unauthorized response. The [Nuget API ](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/nuget_packages.rb )
2020-03-09 05:07:45 -04:00
supports this case.
#### Authorization
There are project and group level permissions for `read_package` , `create_package` , and `destroy_package` . Each
endpoint should
[authorize the requesting user ](https://gitlab.com/gitlab-org/gitlab/blob/398fef1ca26ae2b2c3dc89750f6b20455a1e5507/ee/lib/api/conan_packages.rb )
against the project or group before continuing.
#### Database and handling metadata
2019-05-05 09:57:21 -04:00
The current database model allows you to store a name and a version for each package.
Every time you upload a new package, you can either create a new record of `Package`
or add files to existing record. `PackageFile` should be able to store all file-related
information like the file `name` , `side` , `sha1` , etc.
2019-07-08 23:28:41 -04:00
If there is specific data necessary to be stored for only one package system support,
consider creating a separate metadata model. See `packages_maven_metadata` table
2020-05-08 20:09:39 -04:00
and `Packages::Maven::Metadatum` model as an example for package specific data, and `packages_conan_file_metadata` table
and `Packages::Conan::FileMetadatum` model as an example for package file specific data.
2019-11-14 04:06:15 -05:00
If there is package specific behavior for a given package manager, add those methods to the metadata models and
delegate from the package model.
Note that the existing package UI only displays information within the `packages_packages` and `packages_package_files`
2020-11-20 19:09:36 -05:00
tables. If the data stored in the metadata tables need to be displayed, a ~frontend change is required.
2019-11-14 04:06:15 -05:00
2020-03-09 05:07:45 -04:00
#### File uploads
2019-11-14 04:06:15 -05:00
2020-03-09 05:07:45 -04:00
File uploads should be handled by GitLab Workhorse using object accelerated uploads. What this means is that
2020-11-20 19:09:36 -05:00
the workhorse proxy that checks all incoming requests to GitLab intercept the upload request,
2020-03-09 05:07:45 -04:00
upload the file, and forward a request to the main GitLab codebase only containing the metadata
and file location rather than the file itself. An overview of this process can be found in the
[development documentation ](uploads.md#direct-upload ).
2019-11-14 04:06:15 -05:00
2020-11-20 19:09:36 -05:00
In terms of code, this means a route must be added to the
2020-03-09 05:07:45 -04:00
[GitLab Workhorse project ](https://gitlab.com/gitlab-org/gitlab-workhorse ) for each upload endpoint being added
(instance, group, project). [This merge request ](https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/412/diffs )
demonstrates adding an instance-level endpoint for Conan to workhorse. You can also see the Maven project level endpoint
implemented in the same file.
2019-11-14 04:06:15 -05:00
2020-11-20 19:09:36 -05:00
Once the route has been added, you must add an additional `/authorize` version of the upload endpoint to your API file.
2020-03-09 05:07:45 -04:00
[Here is an example ](https://gitlab.com/gitlab-org/gitlab/blob/398fef1ca26ae2b2c3dc89750f6b20455a1e5507/ee/lib/api/maven_packages.rb#L164 )
of the additional endpoint added for Maven. The `/authorize` endpoint verifies and authorizes the request from workhorse,
then the normal upload endpoint is implemented below, consuming the metadata that workhorse provides in order to
create the package record. Workhorse provides a variety of file metadata such as type, size, and different checksum formats.
2019-11-14 04:06:15 -05:00
2020-03-09 05:07:45 -04:00
For testing purposes, you may want to [enable object storage ](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/object_storage.md )
in your local development environment.
2019-11-14 04:06:15 -05:00
2021-01-14 13:10:59 -05:00
#### Rate Limits on GitLab.com
Package manager clients can make rapid requests that exceed the
[GitLab.com standard API rate limits ](../user/gitlab_com/index.md#gitlabcom-specific-rate-limits ).
This results in a `429 Too Many Requests` error.
We have opened a set of paths to allow higher rate limits. Unless it is not possible,
new package managers should follow these conventions so they can take advantage of the
expanded package rate limit.
These route prefixes guarantee a higher rate limit:
```plaintext
/api/v4/packages/
/api/v4/projects/:project_id/packages/
/api/v4/groups/:group_id/-/packages/
```
2020-03-09 05:07:45 -04:00
### Future Work
2019-11-14 04:06:15 -05:00
2020-11-20 19:09:36 -05:00
While working on the MVC, contributors might find features that are not mandatory for the MVC but can provide a better user experience. It's generally a good idea to keep an eye on those and open issues.
2019-11-14 04:06:15 -05:00
2020-03-09 05:07:45 -04:00
Here are some examples
2019-11-14 04:06:15 -05:00
1. Endpoints required for search
1. Front end updates to display additional package information and metadata
1. Limits on file sizes
1. Tracking for metrics
2020-03-09 05:07:45 -04:00
1. Read more metadata fields from the package to make it available to the front end. For example, it's usual to be able to tag a package. Those tags can be read and saved by backend and then displayed on the packages UI.
1. Endpoints for the upper levels of the [remote hierarchy ](#remote-hierarchy ). This step might need to create a [naming convention ](#naming-conventions )
2019-11-14 04:06:15 -05:00
## Exceptions
This documentation is just guidelines on how to implement a package manager to match the existing structure and logic
already present within GitLab. While the structure is intended to be extendable and flexible enough to allow for
any given package manager, if there is good reason to stray due to the constraints or needs of a given package
manager, then it should be raised and discussed within the implementation issue or merge request to work towards
the most efficient outcome.