Merge branch 'docs/refactor-registry' into 'master'
Move container_registry user docs to new location ## What does this MR do? Move container_registry user docs to new location and refactor them a bit. Part of https://gitlab.com/gitlab-org/gitlab-ce/issues/3349 ## Moving docs to a new location? See the guidelines: http://docs.gitlab.com/ce/development/doc_styleguide.html#changing-document-location - [ ] Make sure the old link is not removed and has its contents replaced with a link to the new location. - [ ] Make sure internal links pointing to the document in question are not broken. - [ ] Search and replace any links referring to old docs in GitLab Rails app, specifically under the `app/views/` directory. - [ ] If working on CE, submit an MR to EE with the changes as well. See merge request !6516
|
@ -6,7 +6,7 @@
|
|||
- [API](api/README.md) Automate GitLab via a simple and powerful API.
|
||||
- [CI/CD](ci/README.md) GitLab Continuous Integration (CI) and Continuous Delivery (CD) getting started, `.gitlab-ci.yml` options, and examples.
|
||||
- [GitLab as OAuth2 authentication service provider](integration/oauth_provider.md). It allows you to login to other applications from GitLab.
|
||||
- [Container Registry](container_registry/README.md) Learn how to use GitLab Container Registry.
|
||||
- [Container Registry](user/project/container_registry.md) Learn how to use GitLab Container Registry.
|
||||
- [GitLab Basics](gitlab-basics/README.md) Find step by step how to start working on your commandline and on GitLab.
|
||||
- [Importing to GitLab](workflow/importing/README.md).
|
||||
- [Importing and exporting projects between instances](user/project/settings/import_export.md).
|
||||
|
|
|
@ -1,42 +1,32 @@
|
|||
# GitLab Container Registry Administration
|
||||
# GitLab Container Registry administration
|
||||
|
||||
> [Introduced][ce-4040] in GitLab 8.8.
|
||||
|
||||
With the Docker Container Registry integrated into GitLab, every project can
|
||||
have its own space to store its Docker images.
|
||||
|
||||
You can read more about Docker Registry at https://docs.docker.com/registry/introduction/.
|
||||
|
||||
---
|
||||
|
||||
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
||||
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
||||
**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
|
||||
> **Notes:**
|
||||
- Container Registry manifest `v1` support was added in GitLab 8.9 to support
|
||||
Docker versions earlier than 1.10.
|
||||
- This document is about the admin guide. To learn how to use GitLab Container
|
||||
Registry [user documentation](../user/project/container_registry.md).
|
||||
|
||||
- [Enable the Container Registry](#enable-the-container-registry)
|
||||
- [Container Registry domain configuration](#container-registry-domain-configuration)
|
||||
- [Configure Container Registry under an existing GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain)
|
||||
- [Configure Container Registry under its own domain](#configure-container-registry-under-its-own-domain)
|
||||
- [Disable Container Registry site-wide](#disable-container-registry-site-wide)
|
||||
- [Disable Container Registry per project](#disable-container-registry-per-project)
|
||||
- [Disable Container Registry for new projects site-wide](#disable-container-registry-for-new-projects-site-wide)
|
||||
- [Container Registry storage path](#container-registry-storage-path)
|
||||
- [Container Registry storage driver](#container-registry-storage-driver)
|
||||
- [Storage limitations](#storage-limitations)
|
||||
- [Changelog](#changelog)
|
||||
With the Container Registry integrated into GitLab, every project can have its
|
||||
own space to store its Docker images.
|
||||
|
||||
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
||||
You can read more about the Container Registry at
|
||||
https://docs.docker.com/registry/introduction/.
|
||||
|
||||
## Enable the Container Registry
|
||||
|
||||
**Omnibus GitLab installations**
|
||||
|
||||
All you have to do is configure the domain name under which the Container
|
||||
Registry will listen to. Read [#container-registry-domain-configuration](#container-registry-domain-configuration)
|
||||
Registry will listen to. Read
|
||||
[#container-registry-domain-configuration](#container-registry-domain-configuration)
|
||||
and pick one of the two options that fits your case.
|
||||
|
||||
>**Note:**
|
||||
The container Registry works under HTTPS by default. Using HTTP is possible
|
||||
The container registry works under HTTPS by default. Using HTTP is possible
|
||||
but not recommended and out of the scope of this document.
|
||||
Read the [insecure Registry documentation][docker-insecure] if you want to
|
||||
implement this.
|
||||
|
@ -47,7 +37,7 @@ implement this.
|
|||
|
||||
If you have installed GitLab from source:
|
||||
|
||||
1. You will have to [install Docker Registry][registry-deploy] by yourself.
|
||||
1. You will have to [install Registry][registry-deploy] by yourself.
|
||||
1. After the installation is complete, you will have to configure the Registry's
|
||||
settings in `gitlab.yml` in order to enable it.
|
||||
1. Use the sample NGINX configuration file that is found under
|
||||
|
@ -80,11 +70,13 @@ where:
|
|||
| `issuer` | This should be the same value as configured in Registry's `issuer`. Read the [token auth configuration documentation][token-config]. |
|
||||
|
||||
>**Note:**
|
||||
GitLab does not ship with a Registry init file. Hence, [restarting GitLab][restart gitlab]
|
||||
will not restart the Registry should you modify its settings. Read the upstream
|
||||
documentation on how to achieve that.
|
||||
A Registry init file is not shipped with GitLab if you install it from source.
|
||||
Hence, [restarting GitLab][restart gitlab] will not restart the Registry should
|
||||
you modify its settings. Read the upstream documentation on how to achieve that.
|
||||
|
||||
The Docker Registry configuration will need `container_registry` as the service and `https://gitlab.example.com/jwt/auth` as the realm:
|
||||
At the absolute minimum, make sure your [Registry configuration][registry-auth]
|
||||
has `container_registry` as the service and `https://gitlab.example.com/jwt/auth`
|
||||
as the realm:
|
||||
|
||||
```
|
||||
auth:
|
||||
|
@ -275,12 +267,6 @@ Registry application itself.
|
|||
|
||||
1. Save the file and [restart GitLab][] for the changes to take effect.
|
||||
|
||||
## Disable Container Registry per project
|
||||
|
||||
If Registry is enabled in your GitLab instance, but you don't need it for your
|
||||
project, you can disable it from your project's settings. Read the user guide
|
||||
on how to achieve that.
|
||||
|
||||
## Disable Container Registry for new projects site-wide
|
||||
|
||||
If the Container Registry is enabled, then it will be available on all new
|
||||
|
@ -436,6 +422,46 @@ storage:
|
|||
enabled: true
|
||||
```
|
||||
|
||||
## Change the registry's internal port
|
||||
|
||||
> **Note:**
|
||||
This is not to be confused with the port that GitLab itself uses to expose
|
||||
the Registry to the world.
|
||||
|
||||
The Registry server listens on localhost at port `5000` by default,
|
||||
which is the address for which the Registry server should accept connections.
|
||||
In the examples below we set the Registry's port to `5001`.
|
||||
|
||||
**Omnibus GitLab**
|
||||
|
||||
1. Open `/etc/gitlab/gitlab.rb` and set `registry['registry_http_addr']`:
|
||||
|
||||
```ruby
|
||||
registry['registry_http_addr'] = "localhost:5001"
|
||||
```
|
||||
|
||||
1. Save the file and [reconfigure GitLab][] for the changes to take effect.
|
||||
|
||||
---
|
||||
|
||||
**Installations from source**
|
||||
|
||||
1. Open the configuration file of your Registry server and edit the
|
||||
[`http:addr`][registry-http-config] value:
|
||||
|
||||
```
|
||||
http
|
||||
addr: localhost:5001
|
||||
```
|
||||
|
||||
1. Save the file and restart the Registry server.
|
||||
|
||||
## Disable Container Registry per project
|
||||
|
||||
If Registry is enabled in your GitLab instance, but you don't need it for your
|
||||
project, you can disable it from your project's settings. Read the user guide
|
||||
on how to achieve that.
|
||||
|
||||
## Storage limitations
|
||||
|
||||
Currently, there is no storage limitation, which means a user can upload an
|
||||
|
@ -455,6 +481,8 @@ configurable in future releases.
|
|||
[docker-insecure]: https://docs.docker.com/registry/insecure/
|
||||
[registry-deploy]: https://docs.docker.com/registry/deploying/
|
||||
[storage-config]: https://docs.docker.com/registry/configuration/#storage
|
||||
[registry-http-config]: https://docs.docker.com/registry/configuration/#http
|
||||
[registry-auth]: https://docs.docker.com/registry/configuration/#auth
|
||||
[token-config]: https://docs.docker.com/registry/configuration/#token
|
||||
[8-8-docs]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-8-stable/doc/administration/container_registry.md
|
||||
[registry-ssl]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/support/nginx/registry-ssl
|
||||
|
|
|
@ -1,98 +1 @@
|
|||
# GitLab Container Registry
|
||||
|
||||
> [Introduced][ce-4040] in GitLab 8.8. Docker Registry manifest
|
||||
`v1` support was added in GitLab 8.9 to support Docker versions earlier than 1.10.
|
||||
|
||||
> **Note:**
|
||||
This document is about the user guide. To learn how to enable GitLab Container
|
||||
Registry across your GitLab instance, visit the
|
||||
[administrator documentation](../administration/container_registry.md).
|
||||
|
||||
With the Docker Container Registry integrated into GitLab, every project can
|
||||
have its own space to store its Docker images.
|
||||
|
||||
You can read more about Docker Registry at https://docs.docker.com/registry/introduction/.
|
||||
|
||||
---
|
||||
|
||||
## Enable the Container Registry for your project
|
||||
|
||||
1. First, ask your system administrator to enable GitLab Container Registry
|
||||
following the [administration documentation](../administration/container_registry.md).
|
||||
If you are using GitLab.com, this is enabled by default so you can start using
|
||||
the Registry immediately.
|
||||
|
||||
1. Go to your project's settings and enable the **Container Registry** feature
|
||||
on your project. For new projects this might be enabled by default. For
|
||||
existing projects you will have to explicitly enable it.
|
||||
|
||||
![Enable Container Registry](img/project_feature.png)
|
||||
|
||||
## Build and push images
|
||||
|
||||
After you save your project's settings, you should see a new link in the
|
||||
sidebar called **Container Registry**. Following this link will get you to
|
||||
your project's Registry panel where you can see how to login to the Container
|
||||
Registry using your GitLab credentials.
|
||||
|
||||
For example if the Registry's URL is `registry.example.com`, the you should be
|
||||
able to login with:
|
||||
|
||||
```
|
||||
docker login registry.example.com
|
||||
```
|
||||
|
||||
Building and publishing images should be a straightforward process. Just make
|
||||
sure that you are using the Registry URL with the namespace and project name
|
||||
that is hosted on GitLab:
|
||||
|
||||
```
|
||||
docker build -t registry.example.com/group/project .
|
||||
docker push registry.example.com/group/project
|
||||
```
|
||||
|
||||
## Use images from GitLab Container Registry
|
||||
|
||||
To download and run a container from images hosted in GitLab Container Registry,
|
||||
use `docker run`:
|
||||
|
||||
```
|
||||
docker run [options] registry.example.com/group/project [arguments]
|
||||
```
|
||||
|
||||
For more information on running Docker containers, visit the
|
||||
[Docker documentation][docker-docs].
|
||||
|
||||
## Control Container Registry from within GitLab
|
||||
|
||||
GitLab offers a simple Container Registry management panel. Go to your project
|
||||
and click **Container Registry** in the left sidebar.
|
||||
|
||||
This view will show you all tags in your project and will easily allow you to
|
||||
delete them.
|
||||
|
||||
![Container Registry panel](img/container_registry.png)
|
||||
|
||||
## Build and push images using GitLab CI
|
||||
|
||||
> **Note:**
|
||||
This feature requires GitLab 8.8 and GitLab Runner 1.2.
|
||||
|
||||
Make sure that your GitLab Runner is configured to allow building Docker images by
|
||||
following the [Using Docker Build](../ci/docker/using_docker_build.md)
|
||||
and [Using the GitLab Container Registry documentation](../ci/docker/using_docker_build.md#using-the-gitlab-container-registry).
|
||||
|
||||
## Limitations
|
||||
|
||||
In order to use a container image from your private project as an `image:` in
|
||||
your `.gitlab-ci.yml`, you have to follow the
|
||||
[Using a private Docker Registry][private-docker]
|
||||
documentation. This workflow will be simplified in the future.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
See [the GitLab Docker registry troubleshooting guide](troubleshooting.md).
|
||||
|
||||
[ce-4040]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4040
|
||||
[docker-docs]: https://docs.docker.com/engine/userguide/intro/
|
||||
[private-docker]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry
|
||||
This document was moved in [user/project/container_registry](../user/project/container_registry.md).
|
||||
|
|
Before Width: | Height: | Size: 218 KiB |
Before Width: | Height: | Size: 243 KiB |
|
@ -1,141 +1 @@
|
|||
# Troubleshooting the GitLab Container Registry
|
||||
|
||||
## Basic Troubleshooting
|
||||
|
||||
1. Check to make sure that the system clock on your Docker client and GitLab server have
|
||||
been synchronized (e.g. via NTP).
|
||||
|
||||
2. If you are using an S3-backed Registry, double check that the IAM
|
||||
permissions and the S3 credentials (including region) are correct. See [the
|
||||
sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/)
|
||||
for more details.
|
||||
|
||||
3. Check the Registry logs (e.g. `/var/log/gitlab/registry/current`) and the GitLab production logs
|
||||
for errors (e.g. `/var/log/gitlab/gitlab-rails/production.log`). You may be able to find clues
|
||||
there.
|
||||
|
||||
## Advanced Troubleshooting
|
||||
|
||||
>**NOTE:** The following section is only recommended for experts.
|
||||
|
||||
Sometimes it's not obvious what is wrong, and you may need to dive deeper into
|
||||
the communication between the Docker client and the Registry to find out
|
||||
what's wrong. We will use a concrete example in the past to illustrate how to
|
||||
diagnose a problem with the S3 setup.
|
||||
|
||||
### Unexpected 403 error during push
|
||||
|
||||
A user attempted to enable an S3-backed Registry. The `docker login` step went
|
||||
fine. However, when pushing an image, the output showed:
|
||||
|
||||
```
|
||||
The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test]
|
||||
dc5e59c14160: Pushing [==================================================>] 14.85 kB
|
||||
03c20c1a019a: Pushing [==================================================>] 2.048 kB
|
||||
a08f14ef632e: Pushing [==================================================>] 2.048 kB
|
||||
228950524c88: Pushing 2.048 kB
|
||||
6a8ecde4cc03: Pushing [==> ] 9.901 MB/205.7 MB
|
||||
5f70bf18a086: Pushing 1.024 kB
|
||||
737f40e80b7f: Waiting
|
||||
82b57dbc5385: Waiting
|
||||
19429b698a22: Waiting
|
||||
9436069b92a3: Waiting
|
||||
error parsing HTTP 403 response body: unexpected end of JSON input: ""
|
||||
```
|
||||
|
||||
This error is ambiguous, as it's not clear whether the 403 is coming from the
|
||||
GitLab Rails application, the Docker Registry, or something else. In this
|
||||
case, since we know that since the login succeeded, we probably need to look
|
||||
at the communication between the client and the Registry.
|
||||
|
||||
The REST API between the Docker client and Registry is [described
|
||||
here](https://docs.docker.com/registry/spec/api/). Normally, one would just
|
||||
use Wireshark or tcpdump to capture the traffic and see where things went
|
||||
wrong. However, since all communication between Docker clients and servers
|
||||
are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even
|
||||
if you know the private key. What can we do instead?
|
||||
|
||||
One way would be to disable HTTPS by setting up an [insecure
|
||||
Registry](https://docs.docker.com/registry/insecure/). This could introduce a
|
||||
security hole and is only recommended for local testing. If you have a
|
||||
production system and can't or don't want to do this, there is another way:
|
||||
use mitmproxy, which stands for Man-in-the-Middle Proxy.
|
||||
|
||||
### mitmproxy
|
||||
|
||||
[mitmproxy](https://mitmproxy.org/) allows you to place a proxy between your
|
||||
client and server to inspect all traffic. One wrinkle is that your system
|
||||
needs to trust the mitmproxy SSL certificates for this to work.
|
||||
|
||||
The following installation instructions assume you are running Ubuntu:
|
||||
|
||||
1. Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html)
|
||||
1. Run `mitmproxy --port 9000` to generate its certificates.
|
||||
Enter <kbd>CTRL</kbd>-<kbd>C</kbd> to quit.
|
||||
1. Install the certificate from `~/.mitmproxy` to your system:
|
||||
|
||||
```sh
|
||||
sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt
|
||||
sudo update-ca-certificates
|
||||
```
|
||||
|
||||
If successful, the output should indicate that a certificate was added:
|
||||
|
||||
```sh
|
||||
Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
|
||||
Running hooks in /etc/ca-certificates/update.d....done.
|
||||
```
|
||||
|
||||
To verify that the certificates are properly installed, run:
|
||||
|
||||
```sh
|
||||
mitmproxy --port 9000
|
||||
```
|
||||
|
||||
This will run mitmproxy on port `9000`. In another window, run:
|
||||
|
||||
```sh
|
||||
curl --proxy http://localhost:9000 https://httpbin.org/status/200
|
||||
```
|
||||
|
||||
If everything is setup correctly, you will see information on the mitmproxy window and
|
||||
no errors from the curl commands.
|
||||
|
||||
### Running the Docker daemon with a proxy
|
||||
|
||||
For Docker to connect through a proxy, you must start the Docker daemon with the
|
||||
proper environment variables. The easiest way is to shutdown Docker (e.g. `sudo initctl stop docker`)
|
||||
and then run Docker by hand. As root, run:
|
||||
|
||||
```sh
|
||||
export HTTP_PROXY="http://localhost:9000"
|
||||
export HTTPS_PROXY="https://localhost:9000"
|
||||
docker daemon --debug
|
||||
```
|
||||
|
||||
This will launch the Docker daemon and proxy all connections through mitmproxy.
|
||||
|
||||
### Running the Docker client
|
||||
|
||||
Now that we have mitmproxy and Docker running, we can attempt to login and push
|
||||
a container image. You may need to run as root to do this. For example:
|
||||
|
||||
```sh
|
||||
docker login s3-testing.myregistry.com:4567
|
||||
docker push s3-testing.myregistry.com:4567/root/docker-test
|
||||
```
|
||||
|
||||
In the example above, we see the following trace on the mitmproxy window:
|
||||
|
||||
![mitmproxy output from Docker](img/mitmproxy-docker.png)
|
||||
|
||||
The above image shows:
|
||||
|
||||
* The initial PUT requests went through fine with a 201 status code.
|
||||
* The 201 redirected the client to the S3 bucket.
|
||||
* The HEAD request to the AWS bucket reported a 403 Unauthorized.
|
||||
|
||||
What does this mean? This strongly suggests that the S3 user does not have the right
|
||||
[permissions to perform a HEAD request](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html).
|
||||
The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/).
|
||||
Once the right permissions were set, the error will go away.
|
||||
This document was moved to [user/project/container_registry](../user/project/container_registry.md).
|
||||
|
|
|
@ -0,0 +1,253 @@
|
|||
# GitLab Container Registry
|
||||
|
||||
> [Introduced][ce-4040] in GitLab 8.8.
|
||||
|
||||
---
|
||||
|
||||
> **Note**
|
||||
Docker Registry manifest `v1` support was added in GitLab 8.9 to support Docker
|
||||
versions earlier than 1.10.
|
||||
>
|
||||
This document is about the user guide. To learn how to enable GitLab Container
|
||||
Registry across your GitLab instance, visit the
|
||||
[administrator documentation](../../administration/container_registry.md).
|
||||
|
||||
With the Docker Container Registry integrated into GitLab, every project can
|
||||
have its own space to store its Docker images.
|
||||
|
||||
You can read more about Docker Registry at https://docs.docker.com/registry/introduction/.
|
||||
|
||||
---
|
||||
|
||||
## Enable the Container Registry for your project
|
||||
|
||||
1. First, ask your system administrator to enable GitLab Container Registry
|
||||
following the [administration documentation](../../administration/container_registry.md).
|
||||
If you are using GitLab.com, this is enabled by default so you can start using
|
||||
the Registry immediately.
|
||||
|
||||
1. Go to your project's settings and enable the **Container Registry** feature
|
||||
on your project. For new projects this might be enabled by default. For
|
||||
existing projects (prior GitLab 8.8), you will have to explicitly enable it.
|
||||
|
||||
![Enable Container Registry](img/container_registry_enable.png)
|
||||
|
||||
1. Hit **Save changes** for the changes to take effect. You should now be able
|
||||
to see the **Registry** link in the project menu.
|
||||
|
||||
![Container Registry tab](img/container_registry_tab.png)
|
||||
|
||||
## Build and push images
|
||||
|
||||
If you visit the **Registry** link under your project's menu, you can see the
|
||||
explicit instructions to login to the Container Registry using your GitLab
|
||||
credentials.
|
||||
|
||||
For example if the Registry's URL is `registry.example.com`, the you should be
|
||||
able to login with:
|
||||
|
||||
```
|
||||
docker login registry.example.com
|
||||
```
|
||||
|
||||
Building and publishing images should be a straightforward process. Just make
|
||||
sure that you are using the Registry URL with the namespace and project name
|
||||
that is hosted on GitLab:
|
||||
|
||||
```
|
||||
docker build -t registry.example.com/group/project .
|
||||
docker push registry.example.com/group/project
|
||||
```
|
||||
|
||||
Your image will be named after the following scheme:
|
||||
|
||||
```
|
||||
<registry URL>/<namespace>/<project>
|
||||
```
|
||||
|
||||
As such, the name of the image is unique, but you can differentiate the images
|
||||
using tags.
|
||||
|
||||
## Use images from GitLab Container Registry
|
||||
|
||||
To download and run a container from images hosted in GitLab Container Registry,
|
||||
use `docker run`:
|
||||
|
||||
```
|
||||
docker run [options] registry.example.com/group/project [arguments]
|
||||
```
|
||||
|
||||
For more information on running Docker containers, visit the
|
||||
[Docker documentation][docker-docs].
|
||||
|
||||
## Control Container Registry from within GitLab
|
||||
|
||||
GitLab offers a simple Container Registry management panel. Go to your project
|
||||
and click **Registry** in the project menu.
|
||||
|
||||
This view will show you all tags in your project and will easily allow you to
|
||||
delete them.
|
||||
|
||||
![Container Registry panel](img/container_registry_panel.png)
|
||||
|
||||
## Build and push images using GitLab CI
|
||||
|
||||
> **Note:**
|
||||
This feature requires GitLab 8.8 and GitLab Runner 1.2.
|
||||
|
||||
Make sure that your GitLab Runner is configured to allow building Docker images by
|
||||
following the [Using Docker Build](../ci/docker/using_docker_build.md)
|
||||
and [Using the GitLab Container Registry documentation](../ci/docker/using_docker_build.md#using-the-gitlab-container-registry).
|
||||
|
||||
## Limitations
|
||||
|
||||
In order to use a container image from your private project as an `image:` in
|
||||
your `.gitlab-ci.yml`, you have to follow the
|
||||
[Using a private Docker Registry][private-docker]
|
||||
documentation. This workflow will be simplified in the future.
|
||||
|
||||
## Troubleshooting the GitLab Container Registry
|
||||
|
||||
### Basic Troubleshooting
|
||||
|
||||
1. Check to make sure that the system clock on your Docker client and GitLab server have
|
||||
been synchronized (e.g. via NTP).
|
||||
|
||||
2. If you are using an S3-backed Registry, double check that the IAM
|
||||
permissions and the S3 credentials (including region) are correct. See [the
|
||||
sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/)
|
||||
for more details.
|
||||
|
||||
3. Check the Registry logs (e.g. `/var/log/gitlab/registry/current`) and the GitLab production logs
|
||||
for errors (e.g. `/var/log/gitlab/gitlab-rails/production.log`). You may be able to find clues
|
||||
there.
|
||||
|
||||
### Advanced Troubleshooting
|
||||
|
||||
>**NOTE:** The following section is only recommended for experts.
|
||||
|
||||
Sometimes it's not obvious what is wrong, and you may need to dive deeper into
|
||||
the communication between the Docker client and the Registry to find out
|
||||
what's wrong. We will use a concrete example in the past to illustrate how to
|
||||
diagnose a problem with the S3 setup.
|
||||
|
||||
#### Unexpected 403 error during push
|
||||
|
||||
A user attempted to enable an S3-backed Registry. The `docker login` step went
|
||||
fine. However, when pushing an image, the output showed:
|
||||
|
||||
```
|
||||
The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test]
|
||||
dc5e59c14160: Pushing [==================================================>] 14.85 kB
|
||||
03c20c1a019a: Pushing [==================================================>] 2.048 kB
|
||||
a08f14ef632e: Pushing [==================================================>] 2.048 kB
|
||||
228950524c88: Pushing 2.048 kB
|
||||
6a8ecde4cc03: Pushing [==> ] 9.901 MB/205.7 MB
|
||||
5f70bf18a086: Pushing 1.024 kB
|
||||
737f40e80b7f: Waiting
|
||||
82b57dbc5385: Waiting
|
||||
19429b698a22: Waiting
|
||||
9436069b92a3: Waiting
|
||||
error parsing HTTP 403 response body: unexpected end of JSON input: ""
|
||||
```
|
||||
|
||||
This error is ambiguous, as it's not clear whether the 403 is coming from the
|
||||
GitLab Rails application, the Docker Registry, or something else. In this
|
||||
case, since we know that since the login succeeded, we probably need to look
|
||||
at the communication between the client and the Registry.
|
||||
|
||||
The REST API between the Docker client and Registry is [described
|
||||
here](https://docs.docker.com/registry/spec/api/). Normally, one would just
|
||||
use Wireshark or tcpdump to capture the traffic and see where things went
|
||||
wrong. However, since all communication between Docker clients and servers
|
||||
are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even
|
||||
if you know the private key. What can we do instead?
|
||||
|
||||
One way would be to disable HTTPS by setting up an [insecure
|
||||
Registry](https://docs.docker.com/registry/insecure/). This could introduce a
|
||||
security hole and is only recommended for local testing. If you have a
|
||||
production system and can't or don't want to do this, there is another way:
|
||||
use mitmproxy, which stands for Man-in-the-Middle Proxy.
|
||||
|
||||
#### mitmproxy
|
||||
|
||||
[mitmproxy](https://mitmproxy.org/) allows you to place a proxy between your
|
||||
client and server to inspect all traffic. One wrinkle is that your system
|
||||
needs to trust the mitmproxy SSL certificates for this to work.
|
||||
|
||||
The following installation instructions assume you are running Ubuntu:
|
||||
|
||||
1. Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html)
|
||||
1. Run `mitmproxy --port 9000` to generate its certificates.
|
||||
Enter <kbd>CTRL</kbd>-<kbd>C</kbd> to quit.
|
||||
1. Install the certificate from `~/.mitmproxy` to your system:
|
||||
|
||||
```sh
|
||||
sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt
|
||||
sudo update-ca-certificates
|
||||
```
|
||||
|
||||
If successful, the output should indicate that a certificate was added:
|
||||
|
||||
```sh
|
||||
Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
|
||||
Running hooks in /etc/ca-certificates/update.d....done.
|
||||
```
|
||||
|
||||
To verify that the certificates are properly installed, run:
|
||||
|
||||
```sh
|
||||
mitmproxy --port 9000
|
||||
```
|
||||
|
||||
This will run mitmproxy on port `9000`. In another window, run:
|
||||
|
||||
```sh
|
||||
curl --proxy http://localhost:9000 https://httpbin.org/status/200
|
||||
```
|
||||
|
||||
If everything is setup correctly, you will see information on the mitmproxy window and
|
||||
no errors from the curl commands.
|
||||
|
||||
#### Running the Docker daemon with a proxy
|
||||
|
||||
For Docker to connect through a proxy, you must start the Docker daemon with the
|
||||
proper environment variables. The easiest way is to shutdown Docker (e.g. `sudo initctl stop docker`)
|
||||
and then run Docker by hand. As root, run:
|
||||
|
||||
```sh
|
||||
export HTTP_PROXY="http://localhost:9000"
|
||||
export HTTPS_PROXY="https://localhost:9000"
|
||||
docker daemon --debug
|
||||
```
|
||||
|
||||
This will launch the Docker daemon and proxy all connections through mitmproxy.
|
||||
|
||||
#### Running the Docker client
|
||||
|
||||
Now that we have mitmproxy and Docker running, we can attempt to login and push
|
||||
a container image. You may need to run as root to do this. For example:
|
||||
|
||||
```sh
|
||||
docker login s3-testing.myregistry.com:4567
|
||||
docker push s3-testing.myregistry.com:4567/root/docker-test
|
||||
```
|
||||
|
||||
In the example above, we see the following trace on the mitmproxy window:
|
||||
|
||||
![mitmproxy output from Docker](img/mitmproxy-docker.png)
|
||||
|
||||
The above image shows:
|
||||
|
||||
* The initial PUT requests went through fine with a 201 status code.
|
||||
* The 201 redirected the client to the S3 bucket.
|
||||
* The HEAD request to the AWS bucket reported a 403 Unauthorized.
|
||||
|
||||
What does this mean? This strongly suggests that the S3 user does not have the right
|
||||
[permissions to perform a HEAD request](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html).
|
||||
The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/).
|
||||
Once the right permissions were set, the error will go away.
|
||||
|
||||
[ce-4040]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4040
|
||||
[docker-docs]: https://docs.docker.com/engine/userguide/intro/
|
||||
[private-docker]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry
|
After Width: | Height: | Size: 5.4 KiB |
After Width: | Height: | Size: 94 KiB |
After Width: | Height: | Size: 7.1 KiB |
Before Width: | Height: | Size: 398 KiB After Width: | Height: | Size: 398 KiB |