Merge branch 'docs/git-submodules-with-ci' into 'master'
Refactor the Git submodules with CI docs Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/22567 See merge request !7856
This commit is contained in:
commit
2adc6568a6
|
@ -21,6 +21,7 @@
|
|||
- [CI services (linked docker containers)](services/README.md)
|
||||
- [CI/CD pipelines settings](../user/project/pipelines/settings.md)
|
||||
- [Review Apps](review_apps/index.md)
|
||||
- [Git submodules](git_submodules.md) Using Git submodules in your CI jobs
|
||||
|
||||
## Breaking changes
|
||||
|
||||
|
|
|
@ -0,0 +1,86 @@
|
|||
# Using Git submodules with GitLab CI
|
||||
|
||||
> **Notes:**
|
||||
- GitLab 8.12 introduced a new [CI build permissions model][newperms] and you
|
||||
are encouraged to upgrade your GitLab instance if you haven't done already.
|
||||
If you are **not** using GitLab 8.12 or higher, you would need to work your way
|
||||
around submodules in order to access the sources of e.g., `gitlab.com/group/project`
|
||||
with the use of [SSH keys](ssh_keys/README.md).
|
||||
- With GitLab 8.12 onward, your permissions are used to evaluate what a CI build
|
||||
can access. More information about how this system works can be found in the
|
||||
[Build permissions model](../user/permissions.md#builds-permissions).
|
||||
- The HTTP(S) Git protocol [must be enabled][gitpro] in your GitLab instance.
|
||||
|
||||
## Configuring the `.gitmodules` file
|
||||
|
||||
If dealing with [Git submodules][gitscm], your project will probably have a file
|
||||
named `.gitmodules`.
|
||||
|
||||
Let's consider the following example:
|
||||
|
||||
1. Your project is located at `https://gitlab.com/secret-group/my-project`.
|
||||
1. To checkout your sources you usually use an SSH address like
|
||||
`git@gitlab.com:secret-group/my-project.git`.
|
||||
1. Your project depends on `https://gitlab.com/group/project`, which you want
|
||||
to include as a submodule.
|
||||
|
||||
If you are using GitLab 8.12+ and your submodule is on the same GitLab server,
|
||||
you must update your `.gitmodules` file to use **relative URLs**.
|
||||
Since Git allows the usage of relative URLs for your `.gitmodules` configuration,
|
||||
this easily allows you to use HTTP(S) for cloning all your CI builds and SSH
|
||||
for all your local checkouts. The `.gitmodules` would look like:
|
||||
|
||||
```ini
|
||||
[submodule "project"]
|
||||
path = project
|
||||
url = ../../group/project.git
|
||||
```
|
||||
|
||||
The above configuration will instruct Git to automatically deduce the URL that
|
||||
should be used when cloning sources. Whether you use HTTP(S) or SSH, Git will use
|
||||
that same channel and it will allow to make all your CI builds use HTTP(S)
|
||||
(because GitLab CI only uses HTTP(S) for cloning your sources), and all your local
|
||||
clones will continue using SSH.
|
||||
|
||||
For all other submodules not located on the same GitLab server, use the full
|
||||
HTTP(S) protocol URL:
|
||||
|
||||
```ini
|
||||
[submodule "project-x"]
|
||||
path = project-x
|
||||
url = https://gitserver.com/group/project-x.git
|
||||
```
|
||||
|
||||
Once `.gitmodules` is correctly configured, you can move on to
|
||||
[configuring your `.gitlab-ci.yml`](#using-git-submodules-in-your-ci-jobs).
|
||||
|
||||
## Using Git submodules in your CI jobs
|
||||
|
||||
There are a few steps you need to take in order to make submodules work
|
||||
correctly with your CI builds:
|
||||
|
||||
1. First, make sure you have used [relative URLs](#configuring-the-gitmodules-file)
|
||||
for the submodules located in the same GitLab server.
|
||||
1. Then, use `git submodule sync/update` in `before_script`:
|
||||
|
||||
```yaml
|
||||
before_script:
|
||||
- git submodule sync --recursive
|
||||
- git submodule update --init --recursive
|
||||
```
|
||||
|
||||
`--recursive` should be used in either both or none (`sync/update`) depending on
|
||||
whether you have recursive submodules.
|
||||
|
||||
The rationale to set the `sync` and `update` in `before_script` is because of
|
||||
the way Git submodules work. On a fresh Runner workspace, Git will set the
|
||||
submodule URL including the token in `.git/config`
|
||||
(or `.git/modules/<submodule>/config`) based on `.gitmodules` and the current
|
||||
remote URL. On subsequent builds on the same Runner, `.git/config` is cached
|
||||
and already contains a full URL for the submodule, corresponding to the previous
|
||||
build, and to **a token from a previous build**. `sync` allows to force updating
|
||||
the full URL.
|
||||
|
||||
[gitpro]: ../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols
|
||||
[gitscm]: https://git-scm.com/book/en/v2/Git-Tools-Submodules "Git submodules documentation"
|
||||
[newperms]: ../user/project/new_ci_build_permissions_model.md
|
|
@ -141,10 +141,9 @@ instance and project. In addition, all admins can use the admin interface under
|
|||
| See events in the system | | | | ✓ |
|
||||
| Admin interface | | | | ✓ |
|
||||
|
||||
### Build permissions
|
||||
|
||||
> Changed in GitLab 8.12.
|
||||
### Builds permissions
|
||||
|
||||
>**Note:**
|
||||
GitLab 8.12 has a completely redesigned build permissions system.
|
||||
Read all about the [new model and its implications][new-mod].
|
||||
|
||||
|
|
|
@ -34,9 +34,9 @@ as created be the pusher (local push or via the UI) and any build created in thi
|
|||
pipeline will have the permissions of the pusher.
|
||||
|
||||
This allows us to make it really easy to evaluate the access for all projects
|
||||
that have Git submodules or are using container images that the pusher would
|
||||
have access too. **The permission is granted only for time that build is running.
|
||||
The access is revoked after the build is finished.**
|
||||
that have [Git submodules][gitsub] or are using container images that the pusher
|
||||
would have access too. **The permission is granted only for time that build is
|
||||
running. The access is revoked after the build is finished.**
|
||||
|
||||
## Types of users
|
||||
|
||||
|
@ -141,7 +141,7 @@ with GitLab 8.12.
|
|||
With the new build permissions model, there is now an easy way to access all
|
||||
dependent source code in a project. That way, we can:
|
||||
|
||||
1. Access a project's Git submodules
|
||||
1. Access a project's [Git submodules][gitsub]
|
||||
1. Access private container images
|
||||
1. Access project's and submodule LFS objects
|
||||
|
||||
|
@ -179,108 +179,8 @@ As a user:
|
|||
|
||||
### Git submodules
|
||||
|
||||
>
|
||||
It often happens that while working on one project, you need to use another
|
||||
project from within it; perhaps it’s a library that a third party developed or
|
||||
you’re developing a project separately and are using it in multiple parent
|
||||
projects.
|
||||
A common issue arises in these scenarios: you want to be able to treat the two
|
||||
projects as separate yet still be able to use one from within the other.
|
||||
>
|
||||
_Excerpt from the [Git website][git-scm] about submodules._
|
||||
|
||||
If dealing with submodules, your project will probably have a file named
|
||||
`.gitmodules`. And this is how it usually looks like:
|
||||
|
||||
```
|
||||
[submodule "tools"]
|
||||
path = tools
|
||||
url = git@gitlab.com/group/tools.git
|
||||
```
|
||||
|
||||
> **Note:**
|
||||
If you are **not** using GitLab 8.12 or higher, you would need to work your way
|
||||
around this issue in order to access the sources of `gitlab.com/group/tools`
|
||||
(e.g., use [SSH keys](../ssh_keys/README.md)).
|
||||
>
|
||||
With GitLab 8.12 onward, your permissions are used to evaluate what a CI build
|
||||
can access. More information about how this system works can be found in the
|
||||
[Build permissions model](../../user/permissions.md#builds-permissions).
|
||||
|
||||
To make use of the new changes, you have to update your `.gitmodules` file to
|
||||
use a relative URL.
|
||||
|
||||
Let's consider the following example:
|
||||
|
||||
1. Your project is located at `https://gitlab.com/secret-group/my-project`.
|
||||
1. To checkout your sources you usually use an SSH address like
|
||||
`git@gitlab.com:secret-group/my-project.git`.
|
||||
1. Your project depends on `https://gitlab.com/group/tools`.
|
||||
1. You have the `.gitmodules` file with above content.
|
||||
|
||||
Since Git allows the usage of relative URLs for your `.gitmodules` configuration,
|
||||
this easily allows you to use HTTP for cloning all your CI builds and SSH
|
||||
for all your local checkouts.
|
||||
|
||||
For example, if you change the `url` of your `tools` dependency, from
|
||||
`git@gitlab.com/group/tools.git` to `../../group/tools.git`, this will instruct
|
||||
Git to automatically deduce the URL that should be used when cloning sources.
|
||||
Whether you use HTTP or SSH, Git will use that same channel and it will allow
|
||||
to make all your CI builds use HTTPS (because GitLab CI uses HTTPS for cloning
|
||||
your sources), and all your local clones will continue using SSH.
|
||||
|
||||
Given the above explanation, your `.gitmodules` file should eventually look
|
||||
like this:
|
||||
|
||||
```
|
||||
[submodule "tools"]
|
||||
path = tools
|
||||
url = ../../group/tools.git
|
||||
```
|
||||
|
||||
However, you have to explicitly tell GitLab CI to clone your submodules as this
|
||||
is not done automatically. You can achieve that by adding a `before_script`
|
||||
section to your `.gitlab-ci.yml`:
|
||||
|
||||
```
|
||||
before_script:
|
||||
- git submodule update --init --recursive
|
||||
|
||||
test:
|
||||
script:
|
||||
- run-my-tests
|
||||
```
|
||||
|
||||
This will make GitLab CI initialize (fetch) and update (checkout) all your
|
||||
submodules recursively.
|
||||
|
||||
If Git does not use the newly added relative URLs but still uses your old URLs,
|
||||
you might need to add `git submodule sync --recursive` to your `.gitlab-ci.yml`,
|
||||
prior to running `git submodule update --init --recursive`. This transfers the
|
||||
changes from your `.gitmodules` file into the `.git` folder, which is kept by
|
||||
runners between runs.
|
||||
|
||||
In case your environment or your Docker image doesn't have Git installed,
|
||||
you have to either ask your Administrator or install the missing dependency
|
||||
yourself:
|
||||
|
||||
```
|
||||
# Debian / Ubuntu
|
||||
before_script:
|
||||
- apt-get update -y
|
||||
- apt-get install -y git-core
|
||||
- git submodule update --init --recursive
|
||||
|
||||
# CentOS / RedHat
|
||||
before_script:
|
||||
- yum install git
|
||||
- git submodule update --init --recursive
|
||||
|
||||
# Alpine
|
||||
before_script:
|
||||
- apk add -U git
|
||||
- git submodule update --init --recursive
|
||||
```
|
||||
To properly configure submodules with GitLab CI, read the
|
||||
[Git submodules documentation][gitsub].
|
||||
|
||||
### Container Registry
|
||||
|
||||
|
@ -310,7 +210,7 @@ test:
|
|||
[build permissions]: ../permissions.md#builds-permissions
|
||||
[comment]: https://gitlab.com/gitlab-org/gitlab-ce/issues/22484#note_16648302
|
||||
[ext]: ../permissions.md#external-users
|
||||
[git-scm]: https://git-scm.com/book/en/v2/Git-Tools-Submodules
|
||||
[gitsub]: ../../ci/git_submodules.md
|
||||
[https]: ../admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols
|
||||
[triggers]: ../../ci/triggers/README.md
|
||||
[update-docs]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update
|
||||
|
|
Loading…
Reference in New Issue