Merge branch 'docs/git-submodules-with-ci' into 'master'
Refactor the Git submodules with CI docs Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/22567 See merge request !7856
This commit is contained in:
commit
2adc6568a6
|
@ -21,6 +21,7 @@
|
||||||
- [CI services (linked docker containers)](services/README.md)
|
- [CI services (linked docker containers)](services/README.md)
|
||||||
- [CI/CD pipelines settings](../user/project/pipelines/settings.md)
|
- [CI/CD pipelines settings](../user/project/pipelines/settings.md)
|
||||||
- [Review Apps](review_apps/index.md)
|
- [Review Apps](review_apps/index.md)
|
||||||
|
- [Git submodules](git_submodules.md) Using Git submodules in your CI jobs
|
||||||
|
|
||||||
## Breaking changes
|
## Breaking changes
|
||||||
|
|
||||||
|
|
|
@ -0,0 +1,86 @@
|
||||||
|
# Using Git submodules with GitLab CI
|
||||||
|
|
||||||
|
> **Notes:**
|
||||||
|
- GitLab 8.12 introduced a new [CI build permissions model][newperms] and you
|
||||||
|
are encouraged to upgrade your GitLab instance if you haven't done already.
|
||||||
|
If you are **not** using GitLab 8.12 or higher, you would need to work your way
|
||||||
|
around submodules in order to access the sources of e.g., `gitlab.com/group/project`
|
||||||
|
with the use of [SSH keys](ssh_keys/README.md).
|
||||||
|
- With GitLab 8.12 onward, your permissions are used to evaluate what a CI build
|
||||||
|
can access. More information about how this system works can be found in the
|
||||||
|
[Build permissions model](../user/permissions.md#builds-permissions).
|
||||||
|
- The HTTP(S) Git protocol [must be enabled][gitpro] in your GitLab instance.
|
||||||
|
|
||||||
|
## Configuring the `.gitmodules` file
|
||||||
|
|
||||||
|
If dealing with [Git submodules][gitscm], your project will probably have a file
|
||||||
|
named `.gitmodules`.
|
||||||
|
|
||||||
|
Let's consider the following example:
|
||||||
|
|
||||||
|
1. Your project is located at `https://gitlab.com/secret-group/my-project`.
|
||||||
|
1. To checkout your sources you usually use an SSH address like
|
||||||
|
`git@gitlab.com:secret-group/my-project.git`.
|
||||||
|
1. Your project depends on `https://gitlab.com/group/project`, which you want
|
||||||
|
to include as a submodule.
|
||||||
|
|
||||||
|
If you are using GitLab 8.12+ and your submodule is on the same GitLab server,
|
||||||
|
you must update your `.gitmodules` file to use **relative URLs**.
|
||||||
|
Since Git allows the usage of relative URLs for your `.gitmodules` configuration,
|
||||||
|
this easily allows you to use HTTP(S) for cloning all your CI builds and SSH
|
||||||
|
for all your local checkouts. The `.gitmodules` would look like:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[submodule "project"]
|
||||||
|
path = project
|
||||||
|
url = ../../group/project.git
|
||||||
|
```
|
||||||
|
|
||||||
|
The above configuration will instruct Git to automatically deduce the URL that
|
||||||
|
should be used when cloning sources. Whether you use HTTP(S) or SSH, Git will use
|
||||||
|
that same channel and it will allow to make all your CI builds use HTTP(S)
|
||||||
|
(because GitLab CI only uses HTTP(S) for cloning your sources), and all your local
|
||||||
|
clones will continue using SSH.
|
||||||
|
|
||||||
|
For all other submodules not located on the same GitLab server, use the full
|
||||||
|
HTTP(S) protocol URL:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[submodule "project-x"]
|
||||||
|
path = project-x
|
||||||
|
url = https://gitserver.com/group/project-x.git
|
||||||
|
```
|
||||||
|
|
||||||
|
Once `.gitmodules` is correctly configured, you can move on to
|
||||||
|
[configuring your `.gitlab-ci.yml`](#using-git-submodules-in-your-ci-jobs).
|
||||||
|
|
||||||
|
## Using Git submodules in your CI jobs
|
||||||
|
|
||||||
|
There are a few steps you need to take in order to make submodules work
|
||||||
|
correctly with your CI builds:
|
||||||
|
|
||||||
|
1. First, make sure you have used [relative URLs](#configuring-the-gitmodules-file)
|
||||||
|
for the submodules located in the same GitLab server.
|
||||||
|
1. Then, use `git submodule sync/update` in `before_script`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
before_script:
|
||||||
|
- git submodule sync --recursive
|
||||||
|
- git submodule update --init --recursive
|
||||||
|
```
|
||||||
|
|
||||||
|
`--recursive` should be used in either both or none (`sync/update`) depending on
|
||||||
|
whether you have recursive submodules.
|
||||||
|
|
||||||
|
The rationale to set the `sync` and `update` in `before_script` is because of
|
||||||
|
the way Git submodules work. On a fresh Runner workspace, Git will set the
|
||||||
|
submodule URL including the token in `.git/config`
|
||||||
|
(or `.git/modules/<submodule>/config`) based on `.gitmodules` and the current
|
||||||
|
remote URL. On subsequent builds on the same Runner, `.git/config` is cached
|
||||||
|
and already contains a full URL for the submodule, corresponding to the previous
|
||||||
|
build, and to **a token from a previous build**. `sync` allows to force updating
|
||||||
|
the full URL.
|
||||||
|
|
||||||
|
[gitpro]: ../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols
|
||||||
|
[gitscm]: https://git-scm.com/book/en/v2/Git-Tools-Submodules "Git submodules documentation"
|
||||||
|
[newperms]: ../user/project/new_ci_build_permissions_model.md
|
|
@ -141,10 +141,9 @@ instance and project. In addition, all admins can use the admin interface under
|
||||||
| See events in the system | | | | ✓ |
|
| See events in the system | | | | ✓ |
|
||||||
| Admin interface | | | | ✓ |
|
| Admin interface | | | | ✓ |
|
||||||
|
|
||||||
### Build permissions
|
### Builds permissions
|
||||||
|
|
||||||
> Changed in GitLab 8.12.
|
|
||||||
|
|
||||||
|
>**Note:**
|
||||||
GitLab 8.12 has a completely redesigned build permissions system.
|
GitLab 8.12 has a completely redesigned build permissions system.
|
||||||
Read all about the [new model and its implications][new-mod].
|
Read all about the [new model and its implications][new-mod].
|
||||||
|
|
||||||
|
|
|
@ -34,9 +34,9 @@ as created be the pusher (local push or via the UI) and any build created in thi
|
||||||
pipeline will have the permissions of the pusher.
|
pipeline will have the permissions of the pusher.
|
||||||
|
|
||||||
This allows us to make it really easy to evaluate the access for all projects
|
This allows us to make it really easy to evaluate the access for all projects
|
||||||
that have Git submodules or are using container images that the pusher would
|
that have [Git submodules][gitsub] or are using container images that the pusher
|
||||||
have access too. **The permission is granted only for time that build is running.
|
would have access too. **The permission is granted only for time that build is
|
||||||
The access is revoked after the build is finished.**
|
running. The access is revoked after the build is finished.**
|
||||||
|
|
||||||
## Types of users
|
## Types of users
|
||||||
|
|
||||||
|
@ -141,7 +141,7 @@ with GitLab 8.12.
|
||||||
With the new build permissions model, there is now an easy way to access all
|
With the new build permissions model, there is now an easy way to access all
|
||||||
dependent source code in a project. That way, we can:
|
dependent source code in a project. That way, we can:
|
||||||
|
|
||||||
1. Access a project's Git submodules
|
1. Access a project's [Git submodules][gitsub]
|
||||||
1. Access private container images
|
1. Access private container images
|
||||||
1. Access project's and submodule LFS objects
|
1. Access project's and submodule LFS objects
|
||||||
|
|
||||||
|
@ -179,108 +179,8 @@ As a user:
|
||||||
|
|
||||||
### Git submodules
|
### Git submodules
|
||||||
|
|
||||||
>
|
To properly configure submodules with GitLab CI, read the
|
||||||
It often happens that while working on one project, you need to use another
|
[Git submodules documentation][gitsub].
|
||||||
project from within it; perhaps it’s a library that a third party developed or
|
|
||||||
you’re developing a project separately and are using it in multiple parent
|
|
||||||
projects.
|
|
||||||
A common issue arises in these scenarios: you want to be able to treat the two
|
|
||||||
projects as separate yet still be able to use one from within the other.
|
|
||||||
>
|
|
||||||
_Excerpt from the [Git website][git-scm] about submodules._
|
|
||||||
|
|
||||||
If dealing with submodules, your project will probably have a file named
|
|
||||||
`.gitmodules`. And this is how it usually looks like:
|
|
||||||
|
|
||||||
```
|
|
||||||
[submodule "tools"]
|
|
||||||
path = tools
|
|
||||||
url = git@gitlab.com/group/tools.git
|
|
||||||
```
|
|
||||||
|
|
||||||
> **Note:**
|
|
||||||
If you are **not** using GitLab 8.12 or higher, you would need to work your way
|
|
||||||
around this issue in order to access the sources of `gitlab.com/group/tools`
|
|
||||||
(e.g., use [SSH keys](../ssh_keys/README.md)).
|
|
||||||
>
|
|
||||||
With GitLab 8.12 onward, your permissions are used to evaluate what a CI build
|
|
||||||
can access. More information about how this system works can be found in the
|
|
||||||
[Build permissions model](../../user/permissions.md#builds-permissions).
|
|
||||||
|
|
||||||
To make use of the new changes, you have to update your `.gitmodules` file to
|
|
||||||
use a relative URL.
|
|
||||||
|
|
||||||
Let's consider the following example:
|
|
||||||
|
|
||||||
1. Your project is located at `https://gitlab.com/secret-group/my-project`.
|
|
||||||
1. To checkout your sources you usually use an SSH address like
|
|
||||||
`git@gitlab.com:secret-group/my-project.git`.
|
|
||||||
1. Your project depends on `https://gitlab.com/group/tools`.
|
|
||||||
1. You have the `.gitmodules` file with above content.
|
|
||||||
|
|
||||||
Since Git allows the usage of relative URLs for your `.gitmodules` configuration,
|
|
||||||
this easily allows you to use HTTP for cloning all your CI builds and SSH
|
|
||||||
for all your local checkouts.
|
|
||||||
|
|
||||||
For example, if you change the `url` of your `tools` dependency, from
|
|
||||||
`git@gitlab.com/group/tools.git` to `../../group/tools.git`, this will instruct
|
|
||||||
Git to automatically deduce the URL that should be used when cloning sources.
|
|
||||||
Whether you use HTTP or SSH, Git will use that same channel and it will allow
|
|
||||||
to make all your CI builds use HTTPS (because GitLab CI uses HTTPS for cloning
|
|
||||||
your sources), and all your local clones will continue using SSH.
|
|
||||||
|
|
||||||
Given the above explanation, your `.gitmodules` file should eventually look
|
|
||||||
like this:
|
|
||||||
|
|
||||||
```
|
|
||||||
[submodule "tools"]
|
|
||||||
path = tools
|
|
||||||
url = ../../group/tools.git
|
|
||||||
```
|
|
||||||
|
|
||||||
However, you have to explicitly tell GitLab CI to clone your submodules as this
|
|
||||||
is not done automatically. You can achieve that by adding a `before_script`
|
|
||||||
section to your `.gitlab-ci.yml`:
|
|
||||||
|
|
||||||
```
|
|
||||||
before_script:
|
|
||||||
- git submodule update --init --recursive
|
|
||||||
|
|
||||||
test:
|
|
||||||
script:
|
|
||||||
- run-my-tests
|
|
||||||
```
|
|
||||||
|
|
||||||
This will make GitLab CI initialize (fetch) and update (checkout) all your
|
|
||||||
submodules recursively.
|
|
||||||
|
|
||||||
If Git does not use the newly added relative URLs but still uses your old URLs,
|
|
||||||
you might need to add `git submodule sync --recursive` to your `.gitlab-ci.yml`,
|
|
||||||
prior to running `git submodule update --init --recursive`. This transfers the
|
|
||||||
changes from your `.gitmodules` file into the `.git` folder, which is kept by
|
|
||||||
runners between runs.
|
|
||||||
|
|
||||||
In case your environment or your Docker image doesn't have Git installed,
|
|
||||||
you have to either ask your Administrator or install the missing dependency
|
|
||||||
yourself:
|
|
||||||
|
|
||||||
```
|
|
||||||
# Debian / Ubuntu
|
|
||||||
before_script:
|
|
||||||
- apt-get update -y
|
|
||||||
- apt-get install -y git-core
|
|
||||||
- git submodule update --init --recursive
|
|
||||||
|
|
||||||
# CentOS / RedHat
|
|
||||||
before_script:
|
|
||||||
- yum install git
|
|
||||||
- git submodule update --init --recursive
|
|
||||||
|
|
||||||
# Alpine
|
|
||||||
before_script:
|
|
||||||
- apk add -U git
|
|
||||||
- git submodule update --init --recursive
|
|
||||||
```
|
|
||||||
|
|
||||||
### Container Registry
|
### Container Registry
|
||||||
|
|
||||||
|
@ -310,7 +210,7 @@ test:
|
||||||
[build permissions]: ../permissions.md#builds-permissions
|
[build permissions]: ../permissions.md#builds-permissions
|
||||||
[comment]: https://gitlab.com/gitlab-org/gitlab-ce/issues/22484#note_16648302
|
[comment]: https://gitlab.com/gitlab-org/gitlab-ce/issues/22484#note_16648302
|
||||||
[ext]: ../permissions.md#external-users
|
[ext]: ../permissions.md#external-users
|
||||||
[git-scm]: https://git-scm.com/book/en/v2/Git-Tools-Submodules
|
[gitsub]: ../../ci/git_submodules.md
|
||||||
[https]: ../admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols
|
[https]: ../admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols
|
||||||
[triggers]: ../../ci/triggers/README.md
|
[triggers]: ../../ci/triggers/README.md
|
||||||
[update-docs]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update
|
[update-docs]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update
|
||||||
|
|
Loading…
Reference in New Issue