gitlab-org--gitlab-foss/doc/development/automatic_ce_ee_merge.md

Automatic CE->EE merge

Commits pushed to CE master are automatically merged into EE master roughly every 5 minutes. Changes are merged using the recursive=ours merge strategy in the context of EE. This means that any merge conflicts are resolved by taking the EE changes and discarding the CE changes. This removes the need for resolving conflicts or reverting changes, at the cost of absolutely requiring EE merge requests to be created whenever a CE merge request causes merge conflicts. Failing to do so can result in changes not making their way into EE.

Always create an EE merge request if there are conflicts

In CI there is a job called ee_compat_check, which checks if a CE MR causes merge conflicts with EE. If this job reports conflicts, you must create an EE merge request. If you are an external contributor you can ask the reviewer to do this for you.

Always merge EE merge requests before their CE counterparts

In order to avoid conflicts in the CE->EE merge, you should always merge the EE version of your CE merge request first, if present.

Failing to do so will lead to CE changes being discarded when merging into EE, if they cause merge conflicts.

Avoiding CE->EE merge conflicts beforehand

To avoid the conflicts beforehand, check out the Guidelines for implementing Enterprise Edition features.

In any case, the CI ee_compat_check job will tell you if you need to open an EE version of your CE merge request.

Conflicts detection in CE merge requests

For each commit (except on master), the ee_compat_check CI job tries to detect if the current branch's changes will conflict during the CE->EE merge.

The job reports what files are conflicting and how to set up a merge request against EE.

How the job works

1. Generates the diff between your branch and current CE master
2. Tries to apply it to current EE master
3. If it applies cleanly, the job succeeds, otherwise...
4. Detects a branch with the ee- prefix or -ee suffix in EE
5. If it exists, generate the diff between this branch and current EE master
6. Tries to apply it to current EE master
7. If it applies cleanly, the job succeeds

In the case where the job fails, it means you should create an ee-<ce_branch> or <ce_branch>-ee branch, push it to EE and open a merge request against EE master. At this point if you retry the failing job in your CE merge request, it should now pass.

Notes:

• This task is not a silver-bullet, its current goal is to bring awareness to developers that their work needs to be ported to EE.
• Community contributors shouldn't be required to submit merge requests against EE, but reviewers should take actions by either creating such EE merge request or asking a GitLab developer to do it before the merge request is merged.
• If you branch is too far behind master, the job will fail. In that case you should rebase your branch upon latest master.
• Code reviews for merge requests often consist of multiple iterations of feedback and fixes. There is no need to update your EE MR after each iteration. Instead, create an EE MR as soon as you see the ee_compat_check job failing. After you receive the final approval from a Maintainer (but before the CE MR is merged) update the EE MR. This helps to identify significant conflicts sooner, but also reduces the number of times you have to resolve conflicts.
• Please remember to always have your EE merge request merged before the CE version.
• You can use git rerere to avoid resolving the same conflicts multiple times.

Cherry-picking from CE to EE

For avoiding merge conflicts, we use a method of creating equivalent branches for CE and EE. If the ee-compat-check job fails, this process is required.

This method only requires that you have cloned both CE and EE into your computer. If you don't have them yet, please go ahead and clone them:

• Clone CE repo: git clone git@gitlab.com:gitlab-org/gitlab-foss.git
• Clone EE repo: git clone git@gitlab.com:gitlab-org/gitlab.git

And the only additional setup we need is to add CE as remote of EE and vice-versa:

• Open two terminal windows, one in CE, and another one in EE:
  • In EE: git remote add ce git@gitlab.com:gitlab-org/gitlab-foss.git
  • In CE: git remote add ee git@gitlab.com:gitlab-org/gitlab.git

That's all setup we need, so that we can cherry-pick a commit from CE to EE, and from EE to CE.

Now, every time you create an MR for CE and EE:

1. Open two terminal windows, one in CE, and another one in EE
2. In the CE terminal:
   1. Create the CE branch, e.g., branch-example
   2. Make your changes and push a commit (commit A)
   3. Create the CE merge request in GitLab
3. In the EE terminal:
   1. Create the EE-equivalent branch ending with -ee, e.g., git checkout -b branch-example-ee
   2. Fetch the CE branch: git fetch ce branch-example
   3. Cherry-pick the commit A: git cherry-pick commit-A-SHA
   4. If Git prompts you to fix the conflicts, do a git status to check which files contain conflicts, fix them, save the files
   5. Add the changes with git add . but DO NOT commit them
   6. Continue cherry-picking: git cherry-pick --continue
   7. Push to EE: git push origin branch-example-ee
4. Create the EE-equivalent MR and link to the CE MR from the description Ports [CE-MR-LINK] to EE
5. Once all the jobs are passing in both CE and EE, you've addressed the feedback from your own team, and got them approved, the merge requests can be merged.
6. When both MRs are ready, the EE merge request will be merged first, and the CE-equivalent will be merged next.

Important notes:

• The commit SHA can be easily found from the GitLab UI. From a merge request, open the tab Commits and click the copy icon to copy the commit SHA.

• To cherry-pick a commit range, such as (A > B > C > D) use:

  git cherry-pick "oldest-commit-SHA^..newest-commit-SHA"
  

  For example, suppose the commit A is the oldest, and its SHA is 4f5e4018c09ed797fdf446b3752f82e46f5af502, and the commit D is the newest, and its SHA is 80e1c9e56783bd57bd7129828ec20b252ebc0538. The cherry-pick command will be:

  git cherry-pick "4f5e4018c09ed797fdf446b3752f82e46f5af502^..80e1c9e56783bd57bd7129828ec20b252ebc0538"
  

• To cherry-pick a merge commit, use the flag -m 1. For example, suppose that the merge commit SHA is 138f5e2f20289bb376caffa0303adb0cac859ce1:

  git cherry-pick -m 1 138f5e2f20289bb376caffa0303adb0cac859ce1
  

• To cherry-pick multiple commits, such as B and D in a range (A > B > C > D), use:

  git cherry-pick commit-B-SHA commit-D-SHA
  

  For example, suppose commit B SHA = 4f5e4018c09ed797fdf446b3752f82e46f5af502, and the commit D SHA = 80e1c9e56783bd57bd7129828ec20b252ebc0538. The cherry-pick command will be:

  git cherry-pick 4f5e4018c09ed797fdf446b3752f82e46f5af502 80e1c9e56783bd57bd7129828ec20b252ebc0538
  

  This case is particularly useful when you have a merge commit in a sequence of commits and you want to cherry-pick all but the merge commit.

• If you push more commits to the CE branch, you can safely repeat the procedure to cherry-pick them to the EE-equivalent branch. You can do that as many times as necessary, using the same CE and EE branches.

• If you submitted the merge request to the CE repo and the ee-compat-check job passed, you are not required to submit the EE-equivalent MR, but it's still recommended. If the job failed, you are required to submit the EE MR so that you can fix the conflicts in EE before merging your changes into CE.

How we run the Automatic CE->EE merge at GitLab

At GitLab, we use the Merge Train project to keep our GitLab EE repository updated with commits from GitLab CE.

We have a mirror of the Merge Train project configured to run an automatic CE->EE merge job every twenty minutes as a scheduled CI job. The configured Merge Train project is only accessible to authorized GitLab staff.

FAQ

How does automatic merging work?

The automatic merging is performed using a project called Merge Train. This project will clone CE and EE master, and merge CE master into EE master using git merge --strategy=recursive --strategy-option=ours. This process runs multiple times per hour.

For more information on the exact implementation you can refer to the source code.

Why merge automatically?

As we work towards continuous deployments and a single repository for both CE and EE, we need to first make sure that all CE changes make their way into EE as fast as possible. Past experiences and data have shown that periodic CE to EE merge requests do not scale, and often take a very long time to complete. For example, in this comment we determined that the average time to close an upstream merge request is around 5 hours, with peaks up to several days. Periodic merge requests are also frustrating to work with, because they often include many changes unrelated to your own changes.

To resolve these problems, we now merge changes using the ours strategy to automatically resolve merge conflicts. This removes the need for resolving conflicts in a periodic merge request, and allows us to merge changes from CE into EE much faster.

My CE merge request caused conflicts after it was merged. What do I do?

If you notice this, you should set up an EE merge request that resolves these conflicts as soon as possible. Failing to do so can lead to your changes not being available in EE, which may break tests. This in turn would prevent us from being able to deploy.

Won't this setup be risky?

No, not if there is an EE merge request for every CE merge request that causes conflicts and that EE merge request is merged first. In the past we may have been a bit more relaxed when it comes to enforcing EE merge requests, but to enable automatic merging we have to start requiring such merge requests even for the smallest conflicts.

Some files I work with often conflict, how can I best deal with this?

If you find you keep running into merge conflicts, consider refactoring the file so that the EE specific changes are not intertwined with CE code. For Ruby code you can do this by moving the EE code to a separate module, which can then be injected into the appropriate classes or modules. See Guidelines for implementing Enterprise Edition features for more information.

---

Return to Development documentation