Update repository mirroring documentation to reflect current state.
BIN
doc/workflow/img/repository_mirroring_force_update.png
Normal file
After Width: | Height: | Size: 45 KiB |
BIN
doc/workflow/img/repository_mirroring_pull_settings_lower.png
Normal file
After Width: | Height: | Size: 57 KiB |
BIN
doc/workflow/img/repository_mirroring_pull_settings_upper.png
Normal file
After Width: | Height: | Size: 63 KiB |
BIN
doc/workflow/img/repository_mirroring_push_settings.png
Normal file
After Width: | Height: | Size: 49 KiB |
|
@ -1,351 +1,394 @@
|
||||||
# Repository mirroring
|
# Repository mirroring
|
||||||
|
|
||||||
Repository mirroring is a way to mirror repositories from external sources.
|
Repository mirroring allows for mirroring of repositories to and from external sources. It can be
|
||||||
It can be used to mirror all branches, tags, and commits that you have
|
used to mirror branches, tags, and commits between repositories.
|
||||||
in your repository.
|
|
||||||
|
|
||||||
Your mirror at GitLab will be updated automatically. You can
|
A repository mirror at GitLab will be updated automatically. You can also manually trigger an update
|
||||||
also manually trigger an update at most once every 5 minutes.
|
at most once every 5 minutes.
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Repository mirroring is very useful when, for some reason, you must use a
|
Repository mirroring is useful when you want to use a repository outside of GitLab.
|
||||||
project from another source.
|
|
||||||
|
|
||||||
There are two kinds of repository mirroring features supported by GitLab:
|
There are two kinds of repository mirroring supported by GitLab:
|
||||||
**push** and **pull**, the latter being only available in GitLab Enterprise Edition.
|
|
||||||
The **push** method mirrors the repository in GitLab to another location.
|
|
||||||
|
|
||||||
Once the mirror repository is updated, all new branches,
|
- Push: for mirroring a GitLab repository to another location.
|
||||||
tags, and commits will be visible in the project's activity feed.
|
- Pull: for mirroring a repository from another location to GitLab. **[STARTER]**
|
||||||
Users with at least [developer access][perms] to the project can also force an
|
|
||||||
immediate update with the click of a button. This button will not be available if
|
|
||||||
the mirror is already being updated or 5 minutes still haven't passed since its last update.
|
|
||||||
|
|
||||||
A few things/limitations to consider:
|
When the mirror repository is updated, all new branches, tags, and commits will be visible in the
|
||||||
|
project's activity feed.
|
||||||
|
|
||||||
- The repository must be accessible over `http://`, `https://`, `ssh://` or `git://`.
|
Users with at least [developer access](../user/permissions.md) to the project can also force an
|
||||||
- If your HTTP repository is not publicly accessible, add authentication
|
immediate update, unless:
|
||||||
information to the URL, like: `https://username@gitlab.company.com/group/project.git`.
|
|
||||||
In some cases, you might need to use a personal access token instead of a
|
- The mirror is already being updated.
|
||||||
password, e.g., you want to mirror to GitHub and have 2FA enabled.
|
- 5 minutes haven't elapsed since its last update.
|
||||||
- The import will time out after 15 minutes. For repositories that take longer
|
|
||||||
use a clone/push combination.
|
|
||||||
- The Git LFS objects will not be synced. You'll need to push/pull them
|
|
||||||
manually.
|
|
||||||
|
|
||||||
## Use cases
|
## Use cases
|
||||||
|
|
||||||
- You migrated to GitLab but still need to keep your project in another source.
|
The following are some possible use cases for repository mirroring:
|
||||||
In that case, you can simply set it up to mirror to GitLab (pull) and all the
|
|
||||||
essential history of commits, tags and branches will be available in your
|
- You migrated to GitLab but still need to keep your project in another source. In that case, you
|
||||||
GitLab instance.
|
can simply set it up to mirror to GitLab (pull) and all the essential history of commits, tags,
|
||||||
- You have old projects in another source that you don't use actively anymore,
|
and branches will be available in your GitLab instance. **[STARTER]**
|
||||||
but don't want to remove for archiving purposes. In that case, you can create
|
- You have old projects in another source that you don't use actively anymore, but don't want to
|
||||||
a push mirror so that your active GitLab repository can push its changes to the
|
remove for archiving purposes. In that case, you can create a push mirror so that your active
|
||||||
old location.
|
GitLab repository can push its changes to the old location.
|
||||||
|
|
||||||
|
## Pushing to a remote repository **[CORE]**
|
||||||
|
|
||||||
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/249) in GitLab Enterprise
|
||||||
|
> Edition 8.7. [Moved to GitLab Core](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18715) in 10.8.
|
||||||
|
|
||||||
|
For an existing project, you can set up push mirroring as follows:
|
||||||
|
|
||||||
|
1. Navigate to your project's **Settings > Repository** and expand the **Mirroring repositories** section.
|
||||||
|
1. Enter a repository URL.
|
||||||
|
1. Select **Push** from the **Mirror direction** dropdown.
|
||||||
|
1. Select an authentication method from the **Authentication method** dropdown, if necessary.
|
||||||
|
1. Check the **Only mirror protected branches** box, if necessary.
|
||||||
|
1. Click the **Mirror repository** button to save the configuration.
|
||||||
|
|
||||||
|
![Repository mirroring push settings screen](img/repository_mirroring_push_settings.png)
|
||||||
|
|
||||||
|
When push mirroring is enabled, only push commits directly to the mirrored repository to prevent the
|
||||||
|
mirror diverging. All changes will end up in the mirrored repository whenever:
|
||||||
|
|
||||||
|
- Commits are pushed to GitLab.
|
||||||
|
- A [forced update](#forcing-an-update) is initiated.
|
||||||
|
|
||||||
|
Changes pushed to files in the repository are automatically pushed to the remote mirror at least:
|
||||||
|
|
||||||
|
- Within five minutes of being received.
|
||||||
|
- Within one minute if **Only mirror protected branches** is enabled.
|
||||||
|
|
||||||
|
In the case of a diverged branch, you will see an error indicated at the **Mirroring repositories**
|
||||||
|
section.
|
||||||
|
|
||||||
|
### Push only protected branches **[CORE]**
|
||||||
|
|
||||||
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3350) in
|
||||||
|
> [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. [Moved to GitLab Core](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18715) in 10.8.
|
||||||
|
|
||||||
|
You can choose to only push your protected branches from GitLab to your remote repository.
|
||||||
|
|
||||||
|
To use this option, check the **Only mirror protected branches** box when creating a repository
|
||||||
|
mirror.
|
||||||
|
|
||||||
|
## Setting up a push mirror from GitLab to GitHub **[CORE]**
|
||||||
|
|
||||||
|
To set up a mirror from GitLab to GitHub, you need to follow these steps:
|
||||||
|
|
||||||
|
1. Create a [GitHub personal access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) with the `public_repo` box checked.
|
||||||
|
1. Fill in the **Git repository URL** field, with the personal access token instead of a password.
|
||||||
|
For example: `https://<GitHubUsername>:<GitHubPersonalAccessToken>@github.com/group/project.git`.
|
||||||
|
1. Click the **Mirror repository** button.
|
||||||
|
1. Wait, or click the update button.
|
||||||
|
|
||||||
## Pulling from a remote repository **[STARTER]**
|
## Pulling from a remote repository **[STARTER]**
|
||||||
|
|
||||||
>[Introduced][ee-51] in GitLab Enterprise Edition 8.2.
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/51) in GitLab Enterprise Edition 8.2.
|
||||||
|
|
||||||
You can set up a repository to automatically have its branches, tags, and commits
|
You can set up a repository to automatically have its branches, tags, and commits updated from an
|
||||||
updated from an upstream repository. This is useful when a repository you're
|
upstream repository.
|
||||||
interested in is located on a different server, and you want to be able to
|
|
||||||
browse its content and its activity using the familiar GitLab interface.
|
|
||||||
|
|
||||||
When creating a new project, you can enable repository mirroring when you choose
|
This is useful when a repository you're interested in is located on a different server, and you want
|
||||||
to import the repository from "Any repo by URL". Enter the full URL of the Git
|
to be able to browse its content and its activity using the familiar GitLab interface.
|
||||||
repository to pull from and click on the **Mirror repository** checkbox.
|
|
||||||
|
|
||||||
![New project](repository_mirroring/repository_mirroring_new_project.png)
|
To configure mirror pulling for an existing project:
|
||||||
|
|
||||||
For an existing project, you can set up mirror pulling by visiting your project's
|
1. Navigate to your project's **Settings > Repository** and expand the **Mirroring repositories**
|
||||||
**Settings ➔ Repository** and searching for the "Pull from a remote repository"
|
section.
|
||||||
section. Check the "Mirror repository" box and hit **Save changes** at the bottom.
|
1. Enter a repository URL.
|
||||||
You have a few options to choose from one being the user who will be the author
|
1. Select **Pull** from the **Mirror direction** dropdown.
|
||||||
of all events in the activity feed that are the result of an update. This user
|
1. Select an authentication method from the **Authentication method** dropdown, if necessary.
|
||||||
needs to have at least [master access][perms] to the project. Another option is
|
1. If necessary, check the following boxes:
|
||||||
whether you want to trigger builds for mirror updates.
|
- **Overwrite diverged branches**.
|
||||||
|
- **Trigger pipelines for mirror updates**.
|
||||||
|
- **Only mirror protected branches**.
|
||||||
|
1. Click the **Mirror repository** button to save the configuration.
|
||||||
|
|
||||||
![Pull settings](repository_mirroring/repository_mirroring_pull_settings.png)
|
![Repository mirroring pull settings screen - upper part](img/repository_mirroring_pull_settings_upper.png)
|
||||||
|
|
||||||
Since the repository on GitLab functions as a mirror of the upstream repository,
|
---
|
||||||
you are advised not to push commits directly to the repository on GitLab.
|
|
||||||
Instead, any commits should be pushed to the upstream repository, and will end
|
|
||||||
up in the GitLab repository automatically within a certain period of time
|
|
||||||
or when a [forced update](#forcing-an-update) is initiated.
|
|
||||||
|
|
||||||
If you do manually update a branch in the GitLab repository, the branch will
|
![Repository mirroring pull settings screen - lower part](img/repository_mirroring_pull_settings_lower.png)
|
||||||
become diverged from upstream, and GitLab will no longer automatically update
|
|
||||||
this branch to prevent any changes from being lost.
|
|
||||||
|
|
||||||
![Diverged branch](repository_mirroring/repository_mirroring_diverged_branch.png)
|
Because GitLab is now set to pull changes from the upstream repository, you should not push commits
|
||||||
|
directly to the repository on GitLab. Instead, any commits should be pushed to the upstream repository.
|
||||||
|
Changes pushed to the upstream repository will be pulled into the GitLab repository, either:
|
||||||
|
|
||||||
### Trigger update using API **[STARTER]**
|
- Automatically within a certain period of time.
|
||||||
|
- When a [forced update](#forcing-an-update) is initiated.
|
||||||
|
|
||||||
>[Introduced][ee-3453] in GitLab Enterprise Edition 10.3.
|
CAUTION: **Caution:**
|
||||||
|
If you do manually update a branch in the GitLab repository, the branch will become diverged from
|
||||||
|
upstream and GitLab will no longer automatically update this branch to prevent any changes from being lost.
|
||||||
|
|
||||||
Pull mirroring uses polling to detect new branches and commits added upstream,
|
### How it works
|
||||||
often many minutes afterwards. If you notify GitLab by [API][pull-api], updates
|
|
||||||
will be pulled immediately.
|
|
||||||
|
|
||||||
Read the [Pull Mirror Trigger API docs][pull-api].
|
Once you activate the pull mirroring feature, the mirror will be inserted into a queue. A scheduler
|
||||||
|
will start every minute and schedule a fixed number of mirrors for update, based on the configured maximum capacity.
|
||||||
|
|
||||||
### Pull only protected branches **[STARTER]**
|
If the mirror updates successfully, it will be enqueued once again with a small backoff period.
|
||||||
|
|
||||||
>[Introduced][ee-3326] in GitLab Enterprise Edition 10.3.
|
If the mirror fails (for example, a branch diverged from upstream), the project's backoff period is
|
||||||
|
increased each time it fails, up to a maximum amount of time.
|
||||||
You can choose to only pull the protected branches from your remote repository to GitLab.
|
|
||||||
|
|
||||||
To use this option go to your project's repository settings page under pull mirror.
|
|
||||||
|
|
||||||
### Overwrite diverged branches **[STARTER]**
|
|
||||||
|
|
||||||
>[Introduced][ee-4559] in GitLab Enterprise Edition 10.6.
|
|
||||||
|
|
||||||
You can choose to always update your local branch with the remote version even
|
|
||||||
if your local version has diverged from the remote.
|
|
||||||
|
|
||||||
To use this option go to your project's repository settings page under pull mirror.
|
|
||||||
|
|
||||||
### Hard failure **[STARTER]**
|
|
||||||
|
|
||||||
>[Introduced][ee-3117] in GitLab Enterprise Edition 10.2.
|
|
||||||
|
|
||||||
Once a mirror gets retried 14 times in a row, it will get marked as hard failed,
|
|
||||||
this will become visible in either the project main dashboard or in the
|
|
||||||
pull mirror settings page.
|
|
||||||
|
|
||||||
![Hard failed mirror main notice](repository_mirroring/repository_mirroring_hard_failed_main.png)
|
|
||||||
|
|
||||||
![Hard failed mirror settings notice](repository_mirroring/repository_mirroring_hard_failed_settings.png)
|
|
||||||
|
|
||||||
When a project is hard failed, it will no longer get picked up for mirroring.
|
|
||||||
A user can resume the project mirroring again by either [forcing an update](#forcing-an-update)
|
|
||||||
or by changing the import URL in repository settings.
|
|
||||||
|
|
||||||
### SSH authentication **[STARTER]**
|
### SSH authentication **[STARTER]**
|
||||||
|
|
||||||
> [Introduced][ee-2551] in GitLab Starter 9.5
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2551) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.5.
|
||||||
|
|
||||||
If you're mirroring over SSH (i.e., an `ssh://` URL), you can authenticate using
|
SSH authentication is mutual:
|
||||||
password-based authentication, just as over HTTPS, but you can also use public
|
|
||||||
key authentication. This is often more secure than password authentication,
|
|
||||||
especially when the source repository supports [Deploy Keys][deploy-key].
|
|
||||||
|
|
||||||
To get started, navigate to **Settings ➔ Repository ➔ Pull from a remote repository**,
|
- You have to prove to the server that you're allowed to access the repository.
|
||||||
enable mirroring (if not already enabled) and enter an `ssh://` URL.
|
- The server also has to prove to *you* that it's who it claims to be.
|
||||||
|
|
||||||
> **NOTE**: SCP-style URLs, e.g., `git@example.com:group/project.git`, are not
|
You provide your credentials as a password or public key. The server that the source repository
|
||||||
supported at this time.
|
resides on provides its credentials as a "host key", the fingerprint of which needs to be verified manually.
|
||||||
|
|
||||||
Entering the URL adds two features to the page - `Fingerprints` and
|
If you're mirroring over SSH (that is, using an `ssh://` URL), you can authenticate using:
|
||||||
`SSH public key authentication`:
|
|
||||||
|
|
||||||
![Pull settings for SSH](repository_mirroring/repository_mirroring_pull_settings_for_ssh.png)
|
- Password-based authentication, just as over HTTPS.
|
||||||
|
- Public key authentication. This is often more secure than password authentication, especially when
|
||||||
|
the source repository supports [Deploy Keys](../ssh/README.md#deploy-keys).
|
||||||
|
|
||||||
SSH authentication is mutual. You have to prove to the server that you're
|
To get started:
|
||||||
allowed to access the repository, but the server also has to prove to *you* that
|
|
||||||
it's who it claims to be. You provide your credentials as a password or public
|
|
||||||
key. The server that the source repository resides on provides its credentials
|
|
||||||
as a "host key", the fingerprint of which needs to be verified manually.
|
|
||||||
|
|
||||||
Press the `Detect host keys` button. GitLab will fetch the host keys from the
|
1. Navigate to your project's **Settings > Repository** and expand the **Mirroring repositories** section.
|
||||||
server, and display the fingerprints to you:
|
1. Enter an `ssh://` URL for mirroring.
|
||||||
|
|
||||||
![Detect SSH host keys](repository_mirroring/repository_mirroring_detect_host_keys.png)
|
NOTE: **Note:**
|
||||||
|
SCP-style URLs (that is, `git@example.com:group/project.git`) are not supported at this time.
|
||||||
|
|
||||||
|
Entering the URL adds two buttons to the page:
|
||||||
|
|
||||||
|
- **Detect host keys**.
|
||||||
|
- **Input host keys manually**.
|
||||||
|
|
||||||
|
If you click the:
|
||||||
|
|
||||||
|
- **Detect host keys** button, GitLab will fetch the host keys from the server and display the fingerprints.
|
||||||
|
- **Input host keys manually** button, a field is displayed where you can paste in host keys.
|
||||||
|
|
||||||
You now need to verify that the fingerprints are those you expect. GitLab.com
|
You now need to verify that the fingerprints are those you expect. GitLab.com
|
||||||
and other code hosting sites publish their fingerprints in the open for you
|
and other code hosting sites publish their fingerprints in the open for you
|
||||||
to check:
|
to check:
|
||||||
|
|
||||||
* [AWS CodeCommit](http://docs.aws.amazon.com/codecommit/latest/userguide/regions.html#regions-fingerprints)
|
- [AWS CodeCommit](http://docs.aws.amazon.com/codecommit/latest/userguide/regions.html#regions-fingerprints)
|
||||||
* [Bitbucket](https://confluence.atlassian.com/bitbucket/use-the-ssh-protocol-with-bitbucket-cloud-221449711.html#UsetheSSHprotocolwithBitbucketCloud-KnownhostorBitbucket%27spublickeyfingerprints)
|
- [Bitbucket](https://confluence.atlassian.com/bitbucket/use-the-ssh-protocol-with-bitbucket-cloud-221449711.html#UsetheSSHprotocolwithBitbucketCloud-KnownhostorBitbucket%27spublickeyfingerprints)
|
||||||
* [GitHub](https://help.github.com/articles/github-s-ssh-key-fingerprints/)
|
- [GitHub](https://help.github.com/articles/github-s-ssh-key-fingerprints/)
|
||||||
* [GitLab.com](https://about.gitlab.com/gitlab-com/settings/#ssh-host-keys-fingerprints)
|
- [GitLab.com](https://about.gitlab.com/gitlab-com/settings/#ssh-host-keys-fingerprints)
|
||||||
* [Launchpad](https://help.launchpad.net/SSHFingerprints)
|
- [Launchpad](https://help.launchpad.net/SSHFingerprints)
|
||||||
* [Savannah](http://savannah.gnu.org/maintenance/SshAccess/)
|
- [Savannah](http://savannah.gnu.org/maintenance/SshAccess/)
|
||||||
* [SourceForge](https://sourceforge.net/p/forge/documentation/SSH%20Key%20Fingerprints/)
|
- [SourceForge](https://sourceforge.net/p/forge/documentation/SSH%20Key%20Fingerprints/)
|
||||||
|
|
||||||
Other providers will vary. If you're running on-premises GitLab, or otherwise
|
Other providers will vary. If you're running self-managed GitLab, or otherwise
|
||||||
have access to the source server, you can securely gather the key fingerprints:
|
have access to the source server, you can securely gather the key fingerprints:
|
||||||
|
|
||||||
```
|
```sh
|
||||||
$ cat /etc/ssh/ssh_host*pub | ssh-keygen -E md5 -l -f -
|
$ cat /etc/ssh/ssh_host*pub | ssh-keygen -E md5 -l -f -
|
||||||
256 MD5:f4:28:9f:23:99:15:21:1b:bf:ed:1f:8e:a0:76:b2:9d root@example.com (ECDSA)
|
256 MD5:f4:28:9f:23:99:15:21:1b:bf:ed:1f:8e:a0:76:b2:9d root@example.com (ECDSA)
|
||||||
256 MD5:e6:eb:45:8a:3c:59:35:5f:e9:5b:80:12:be:7e:22:73 root@example.com (ED25519)
|
256 MD5:e6:eb:45:8a:3c:59:35:5f:e9:5b:80:12:be:7e:22:73 root@example.com (ED25519)
|
||||||
2048 MD5:3f:72:be:3d:62:03:5c:62:83:e8:6e:14:34:3a:85:1d root@example.com (RSA)
|
2048 MD5:3f:72:be:3d:62:03:5c:62:83:e8:6e:14:34:3a:85:1d root@example.com (RSA)
|
||||||
```
|
```
|
||||||
|
|
||||||
(You may need to exclude `-E md5` for some older versions of SSH).
|
NOTE: **Note:**
|
||||||
|
You may need to exclude `-E md5` for some older versions of SSH.
|
||||||
|
|
||||||
If you're an SSH expert and already have a `known_hosts` file you'd like to use
|
When pulling changes from the source repository, GitLab will now check that at least one of the stored
|
||||||
unaltered, then you can skip these steps. Just press the "Show advanced" button
|
host keys matches before connecting. This can prevent malicious code from being injected into your
|
||||||
and paste in the file contents:
|
mirror, or your password being stolen.
|
||||||
|
|
||||||
![Advanced SSH host key management](repository_mirroring/repository_mirroring_pull_advanced_host_keys.png)
|
### SSH public key authentication
|
||||||
|
|
||||||
Once you've **carefully verified** that all the fingerprints match your trusted
|
To use SSH public key authentication, you'll also need to choose that option from the **Authentication method**
|
||||||
source, you can press `Save changes`. This will record the host keys, along with
|
dropdown. GitLab will generate a 4096-bit RSA key and display the public component of that key to you.
|
||||||
the person who verified them (you!) and the date:
|
|
||||||
|
|
||||||
![SSH host keys submitted](repository_mirroring/repository_mirroring_ssh_host_keys_verified.png)
|
You then need to add the public SSH key to the source repository configuration. If:
|
||||||
|
|
||||||
When pulling changes from the source repository, GitLab will now check that at
|
- The source is hosted on GitLab, you should add the public SSH key as a [Deploy Key](../ssh/README.md#deploy-keys).
|
||||||
least one of the stored host keys matches before connecting. This can prevent
|
- The source is hosted elsewhere, you may need to add the key to your user's `authorized_keys` file.
|
||||||
malicious code from being injected into your mirror, or your password being
|
Paste the entire public SSH key into the file on its own line and save it.
|
||||||
stolen!
|
|
||||||
|
|
||||||
To use SSH public key authentication, you'll also need to choose that option
|
Once the public key is set up on the source repository, click the **Mirror repository** button and
|
||||||
from the authentication methods dropdown. GitLab will generate a 4096-bit RSA
|
your mirror will begin working.
|
||||||
key and display the public component of that key to you:
|
|
||||||
|
|
||||||
![SSH public key authentication](repository_mirroring/repository_mirroring_ssh_public_key_authentication.png)
|
If you need to change the key at any time, you can click the **Regenerate key** button to do so. You'll have to update the source repository with the new key to keep the mirror running.
|
||||||
|
|
||||||
You then need to add the public SSH key to the source repository configuration.
|
### Overwrite diverged branches **[STARTER]**
|
||||||
If the source is hosted on GitLab, you should add it as a [Deploy Key][deploy-key].
|
|
||||||
Other sources may require you to add the key to your user's `authorized_keys`
|
|
||||||
file - just paste the entire `ssh-rsa AAA.... user@host` block into the file on
|
|
||||||
its own line and save it.
|
|
||||||
|
|
||||||
Once the public key is set up on the source repository, press `Save changes` and your
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4559) in
|
||||||
mirror will begin working.
|
> [GitLab Starter](https://about.gitlab.com/pricing/) 10.6.
|
||||||
|
|
||||||
If you need to change the key at any time, you can press the `Regenerate key`
|
You can choose to always update your local branches with remote versions, even if they have
|
||||||
button to do so. You'll have to update the source repository with the new key
|
diverged from the remote.
|
||||||
to keep the mirror running.
|
|
||||||
|
|
||||||
### How it works
|
CAUTION: **Caution:**
|
||||||
|
For mirrored branches, enabling this option results in the loss of local changes.
|
||||||
|
|
||||||
Once you activate the pull mirroring feature, the mirror will be inserted into
|
To use this option, check the **Overwrite diverged branches** box when creating a repository mirror.
|
||||||
a queue. A scheduler will start every minute and schedule a fixed amount of
|
|
||||||
mirrors for update, based on the configured maximum capacity.
|
|
||||||
|
|
||||||
If the mirror successfully updates it will be enqueued once again with a small
|
### Only mirror protected branches **[STARTER]**
|
||||||
backoff period.
|
|
||||||
|
|
||||||
If the mirror fails (eg: branch diverged from upstream), the project's backoff
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3326) in
|
||||||
period will be penalized each time it fails up to a maximum amount of time.
|
> [GitLab Starter](https://about.gitlab.com/pricing/) 10.3.
|
||||||
|
|
||||||
## Pushing to a remote repository
|
You can choose to pull mirror only the protected branches from your remote repository to GitLab.
|
||||||
|
Non-protected branches are not mirrored and can diverge.
|
||||||
|
|
||||||
>[Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/249) in
|
To use this option, check the **Only mirror protected branches** box when creating a repository mirror.
|
||||||
GitLab Enterprise Edition 8.7. [Moved to GitLab Community Edition][ce-18715] in 10.8.
|
|
||||||
|
|
||||||
For an existing project, you can set up push mirror from your project's
|
### Hard failure **[STARTER]**
|
||||||
**Settings ➔ Repository** and searching for the "Push to a remote repository"
|
|
||||||
section. Check the "Remote mirror repository" box and fill in the Git URL of
|
|
||||||
the repository to push to. Click **Save changes** for the changes to take
|
|
||||||
effect.
|
|
||||||
|
|
||||||
![Push settings](repository_mirroring/repository_mirroring_push_settings.png)
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3117) in
|
||||||
|
> [GitLab Starter](https://about.gitlab.com/pricing/) 10.2.
|
||||||
|
|
||||||
When push mirroring is enabled, you are advised not to push commits directly
|
Once the mirroring process is unsuccessfully retried 14 times in a row, it will get marked as hard
|
||||||
to the mirrored repository to prevent the mirror diverging.
|
failed. This will become visible in either the:
|
||||||
All changes will end up in the mirrored repository whenever commits
|
|
||||||
are pushed to GitLab, or when a [forced update](#forcing-an-update) is
|
|
||||||
initiated.
|
|
||||||
|
|
||||||
Pushes into GitLab are automatically pushed to the remote mirror at least once
|
- Project's main dashboard.
|
||||||
every 5 minutes after they are received or once every minute if **push only
|
- Pull mirror settings page.
|
||||||
protected branches** is enabled.
|
|
||||||
|
|
||||||
In case of a diverged branch, you will see an error indicated at the **Mirror
|
When a project is hard failed, it will no longer get picked up for mirroring. A user can resume the
|
||||||
repository** settings.
|
project mirroring again by [Forcing an update](#forcing-an-update).
|
||||||
|
|
||||||
![Diverged branch](
|
### Trigger update using API **[STARTER]**
|
||||||
repository_mirroring/repository_mirroring_diverged_branch_push.png)
|
|
||||||
|
|
||||||
### Push only protected branches
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3453) in
|
||||||
|
[GitLab Starter](https://about.gitlab.com/pricing/) 10.3.
|
||||||
|
|
||||||
>[Introduced][ee-3350] in GitLab Enterprise Edition 10.3. [Moved to GitLab Community Edition][ce-18715] in 10.8.
|
Pull mirroring uses polling to detect new branches and commits added upstream, often minutes
|
||||||
|
afterwards. If you notify GitLab by [API](https://docs.gitlab.com/ee/api/projects.html#start-the-pull-mirroring-process-for-a-project),
|
||||||
|
updates will be pulled immediately.
|
||||||
|
|
||||||
You can choose to only push your protected branches from GitLab to your remote repository.
|
For more information, see [Start the pull mirroring process for a Project](https://docs.gitlab.com/ee/api/projects.html#start-the-pull-mirroring-process-for-a-project).
|
||||||
|
|
||||||
To use this option go to your project's repository settings page under push mirror.
|
## Forcing an update **[CORE]**
|
||||||
|
|
||||||
## Setting up a push mirror from GitLab to GitHub
|
While mirrors are scheduled to update automatically, you can always force an update by using the
|
||||||
|
update button which is available on the **Mirroring repositories** section of the **Repository Settings** page.
|
||||||
|
|
||||||
To set up a mirror from GitLab to GitHub, you need to follow these steps:
|
![Repository mirroring force update user interface](img/repository_mirroring_force_update.png)
|
||||||
|
|
||||||
1. Create a [GitHub personal access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) with the "public_repo" box checked:
|
## Bidirectional mirroring **[STARTER]**
|
||||||
|
|
||||||
![edit personal access token GitHub](repository_mirroring/repository_mirroring_github_edit_personal_access_token.png)
|
CAUTION: **Caution:**
|
||||||
|
Bidirectional mirroring may cause conflicts.
|
||||||
|
|
||||||
1. Fill in the "Git repository URL" with the personal access token replacing the password `https://GitHubUsername:GitHubPersonalAccessToken@github.com/group/project.git`:
|
If you configure a GitLab repository to both pull from, and push to, the same remote source, there
|
||||||
|
is no guarantee that either repository will update correctly. If you set up a repository for
|
||||||
|
bidirectional mirroring, you should prepare for the likely conflicts by deciding who will resolve
|
||||||
|
them and how they will be resolved.
|
||||||
|
|
||||||
![push to remote repo](repository_mirroring/repository_mirroring_gitlab_push_to_a_remote_repository.png)
|
Rewriting any mirrored commit on either remote will cause conflicts and mirroring to fail. This can
|
||||||
|
be prevented by:
|
||||||
|
|
||||||
1. Save
|
- [Pulling only protected branches](#pull-only-protected-branches).
|
||||||
1. And either wait or trigger the "Update Now" button:
|
- [Pushing only protected branches](#push-only-protected-branches).
|
||||||
|
|
||||||
![update now](repository_mirroring/repository_mirroring_gitlab_push_to_a_remote_repository_update_now.png)
|
You should [protect the branches](../user/project/protected_branches.md) you wish to mirror on both
|
||||||
|
remotes to prevent conflicts caused by rewriting history.
|
||||||
|
|
||||||
## Forcing an update
|
Bidirectional mirroring also creates a race condition where commits made close together to the same
|
||||||
|
branch causes conflicts. The race condition can be mitigated by reducing the mirroring delay by using
|
||||||
|
a [Push event webhook](../user/project/integrations/webhooks.md#push-events) to trigger an immediate
|
||||||
|
pull to GitLab. Push mirroring from GitLab is rate limited to once per minute when only push mirroring
|
||||||
|
protected branches.
|
||||||
|
|
||||||
While mirrors are scheduled to update automatically, you can always force an update
|
### Preventing conflicts using a `pre-receive` hook
|
||||||
by using the **Update now** button which is exposed in various places:
|
|
||||||
|
|
||||||
- in the commits page
|
> **Warning:** The solution proposed will negatively impact the performance of
|
||||||
- in the branches page
|
> Git push operations because they will be proxied to the upstream Git
|
||||||
- in the tags page
|
> repository.
|
||||||
- in the **Mirror repository** settings page
|
|
||||||
|
|
||||||
## Bidirectional mirroring
|
A server-side `pre-receive` hook can be used to prevent the race condition
|
||||||
|
described above by only accepting the push after first pushing the commit to
|
||||||
|
the upstream Git repository. In this configuration one Git repository acts as
|
||||||
|
the authoritative upstream, and the other as downstream. The `pre-receive` hook
|
||||||
|
will be installed on the downstream repository.
|
||||||
|
|
||||||
|
Read about [configuring custom Git hooks](../administration/custom_hooks.md) on the GitLab server.
|
||||||
|
|
||||||
|
A sample `pre-receive` hook is provided below.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
|
||||||
|
# --- Assume only one push mirror target
|
||||||
|
# Push mirroring remotes are named `remote_mirror_<id>`, this finds the first remote and uses that.
|
||||||
|
TARGET_REPO=$(git remote | grep -m 1 remote_mirror)
|
||||||
|
|
||||||
|
proxy_push()
|
||||||
|
{
|
||||||
|
# --- Arguments
|
||||||
|
OLDREV=$(git rev-parse $1)
|
||||||
|
NEWREV=$(git rev-parse $2)
|
||||||
|
REFNAME="$3"
|
||||||
|
|
||||||
|
# --- Pattern of branches to proxy pushes
|
||||||
|
whitelisted=$(expr "$branch" : "\(master\)")
|
||||||
|
|
||||||
|
case "$refname" in
|
||||||
|
refs/heads/*)
|
||||||
|
branch=$(expr "$refname" : "refs/heads/\(.*\)")
|
||||||
|
|
||||||
|
if [ "$whitelisted" = "$branch" ]; then
|
||||||
|
error="$(git push --quiet $TARGET_REPO $NEWREV:$REFNAME 2>&1)"
|
||||||
|
fail=$?
|
||||||
|
|
||||||
|
if [ "$fail" != "0" ]; then
|
||||||
|
echo >&2 ""
|
||||||
|
echo >&2 " Error: updates were rejected by upstream server"
|
||||||
|
echo >&2 " This is usually caused by another repository pushing changes"
|
||||||
|
echo >&2 " to the same ref. You may want to first integrate remote changes"
|
||||||
|
echo >&2 ""
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# Allow dual mode: run from the command line just like the update hook, or
|
||||||
|
# if no arguments are given then run as a hook script
|
||||||
|
if [ -n "$1" -a -n "$2" -a -n "$3" ]; then
|
||||||
|
# Output to the terminal in command line mode - if someone wanted to
|
||||||
|
# resend an email; they could redirect the output to sendmail
|
||||||
|
# themselves
|
||||||
|
PAGER= proxy_push $2 $3 $1
|
||||||
|
else
|
||||||
|
# Push is proxied upstream one ref at a time. Because of this it is possible
|
||||||
|
# for some refs to succeed, and others to fail. This will result in a failed
|
||||||
|
# push.
|
||||||
|
while read oldrev newrev refname
|
||||||
|
do
|
||||||
|
proxy_push $oldrev $newrev $refname
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
### Mirroring with Perforce Helix via Git Fusion **[STARTER]**
|
||||||
|
|
||||||
CAUTION: **Warning:**
|
CAUTION: **Warning:**
|
||||||
There is no bidirectional support without conflicts. If you
|
Bidirectional mirroring should not be used as a permanent configuration. Refer to
|
||||||
configure a repository to pull and push to a second remote, there is no
|
[Migrating from Perforce Helix](../user/project/import/perforce.md) for alternative migration approaches.
|
||||||
guarantee that it will update correctly on both remotes. If you configure
|
|
||||||
a repository for bidirectional mirroring, you should consider when conflicts
|
|
||||||
occur who and how they will be resolved.
|
|
||||||
|
|
||||||
Rewriting any mirrored commit on either remote will cause conflicts and
|
[Git Fusion](https://www.perforce.com/video-tutorials/git-fusion-overview) provides a Git interface
|
||||||
mirroring to fail. This can be prevented by [only pulling protected branches](
|
to [Perforce Helix](https://www.perforce.com/products) which can be used by GitLab to bidirectionally
|
||||||
#pull-only-protected-branches) and [only pushing protected branches](
|
mirror projects with GitLab. This may be useful in some situations when migrating from Perforce Helix
|
||||||
#push-only-protected-branches). You should protect the branches you wish to
|
to GitLab where overlapping Perforce Helix workspaces cannot be migrated simultaneously to GitLab.
|
||||||
mirror on both remotes to prevent conflicts caused by rewriting history.
|
|
||||||
|
|
||||||
Bidirectional mirroring also creates a race condition where commits to the same
|
If using mirroring with Perforce Helix, you should only mirror protected branches. Perforce Helix
|
||||||
branch in close proximity will cause conflicts. The race condition can be
|
will reject any pushes that rewrite history. Only the fewest number of branches should be mirrored
|
||||||
mitigated by reducing the mirroring delay by using a Push event webhook to
|
due to the performance limitations of Git Fusion.
|
||||||
trigger an immediate pull to GitLab. Push mirroring from GitLab is rate limited
|
|
||||||
to once per minute when only push mirroring protected branches.
|
|
||||||
|
|
||||||
It may be possible to implement a locking mechanism using the server-side
|
When configuring mirroring with Perforce Helix via Git Fusion, the following Git Fusion
|
||||||
`pre-receive` hook to prevent the race condition. Read about [configuring
|
settings are recommended:
|
||||||
custom Git hooks][hooks] on the GitLab server.
|
|
||||||
|
|
||||||
### Mirroring with Perforce via GitFusion
|
- `change-pusher` should be disabled. Otherwise, every commit will be rewritten as being committed
|
||||||
|
by the mirroring account, rather than being mapped to existing Perforce Helix users or the `unknown_git` user.
|
||||||
|
- `unknown_git` user will be used as the commit author if the GitLab user does not exist in
|
||||||
|
Perforce Helix.
|
||||||
|
|
||||||
CAUTION: **Warning:**
|
Read about [Git Fusion settings on Perforce.com](https://www.perforce.com/perforce/doc.current/manuals/git-fusion/Content/Git-Fusion/section_zdp_zz1_3l.html).
|
||||||
Bidirectional mirroring should not be used as a permanent
|
|
||||||
configuration. There is no bidirectional mirroring without conflicts.
|
|
||||||
Refer to [Migrating from Perforce Helix][perforce] for alternative migration
|
|
||||||
approaches.
|
|
||||||
|
|
||||||
GitFusion provides a Git interface to Perforce which can be used by GitLab to
|
|
||||||
bidirectionally mirror projects with GitLab. This may be useful in some
|
|
||||||
situations when migrating from Perforce to GitLab where overlapping Perforce
|
|
||||||
workspaces cannot be migrated simultaneously to GitLab.
|
|
||||||
|
|
||||||
If using mirroring with Perforce you should only mirror protected branches.
|
|
||||||
Perforce will reject any pushes that rewrite history. It is recommended that
|
|
||||||
only the fewest number of branches are mirrored due to the performance
|
|
||||||
limitations of GitFusion.
|
|
||||||
|
|
||||||
[ee-51]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/51
|
|
||||||
[ee-2551]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2551
|
|
||||||
[ee-3117]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3117
|
|
||||||
[ee-3326]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3326
|
|
||||||
[ee-3350]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3350
|
|
||||||
[ee-3453]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3453
|
|
||||||
[ee-4559]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4559
|
|
||||||
[ce-18715]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18715
|
|
||||||
[perms]: ../user/permissions.md
|
|
||||||
[hooks]: ../administration/custom_hooks.md
|
|
||||||
[deploy-key]: ../ssh/README.md#deploy-keys
|
|
||||||
[webhook]: ../user/project/integrations/webhooks.md#push-events
|
|
||||||
[pull-api]: ../api/projects.md#start-the-pull-mirroring-process-for-a-project
|
|
||||||
[perforce]: ../user/project/import/perforce.md
|
|
||||||
|
|
Before Width: | Height: | Size: 22 KiB |
Before Width: | Height: | Size: 22 KiB |
Before Width: | Height: | Size: 9.3 KiB |
Before Width: | Height: | Size: 20 KiB |
Before Width: | Height: | Size: 16 KiB |
Before Width: | Height: | Size: 16 KiB |
Before Width: | Height: | Size: 15 KiB |
Before Width: | Height: | Size: 16 KiB |
Before Width: | Height: | Size: 20 KiB |
Before Width: | Height: | Size: 40 KiB |
Before Width: | Height: | Size: 39 KiB |
Before Width: | Height: | Size: 26 KiB |
Before Width: | Height: | Size: 18 KiB |
Before Width: | Height: | Size: 9.1 KiB |
Before Width: | Height: | Size: 27 KiB |