gitlab-org--gitlab-foss/doc/migrate_ci_to_ce

Migrate GitLab CI to GitLab CE or EE

Beginning with version 8.0 of GitLab Community Edition (CE) and Enterprise Edition (EE), GitLab CI is no longer its own application, but is instead built into the CE and EE applications.

This guide will detail the process of migrating your CI installation and data into your GitLab CE or EE installation.

We recommend that you read through the entire migration process in this document before beginning.

Overview

In this document we assume you have a GitLab server and a GitLab CI server. It does not matter if these are the same machine.

The migration consists of three parts: updating GitLab and GitLab CI, moving data, and redirecting traffic.

Please note that CI builds triggered on your GitLab server in the time between updating to 8.0 and finishing the migration will be lost.

Before upgrading

• (1) Make sure that the backup script on both servers can connect to the database.

# CI server
# Omnibus
sudo gitlab-ci-rake backup:create

# Source
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production

Also check on your GitLab server.

# GitLab server
# Omnibus
sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads

# Source
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads

If this fails you need to fix it before upgrading to 8.0. Also see https://about.gitlab.com/getting-help/

• (2) Check what databases you use on your GitLab server and your CI server. Look for the 'adapter:' line. If your CI server and your GitLab server use the same database adapter no special care is needed. If your CI server uses MySQL and your GitLab server uses PostgreSQL you need to pass a special option during the 'Moving data' part. If your CI server uses PostgreSQL and your GitLab server uses MySQL you cannot migrate your CI data to GitLab 8.0.

# CI server
# Omnibus
sudo gitlab-ci-rake env:info

# Source
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rake env:info RAILS_ENV=production

# GitLab server
# Omnibus
sudo gitlab-rake gitlab:env:info

# Source
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production

• (3) Decide where to store CI build traces on GitLab server. GitLab CI uses files on disk to store CI build traces. The default path for these build traces is /var/opt/gitlab/gitlab-ci/builds (Omnibus) or /home/git/gitlab/builds (Source). If you are storing your repository data in a special location, or if you are using NFS, you should make sure that you store build traces on the same storage as your Git repositories.

Upgrading

From this point on, GitLab CI will be unavailable for your end users.

• (1) First upgrade your GitLab server to version 8.0: https://about.gitlab.com/update/

• (2) After you update, go to the admin panel and temporarily disable CI. As an administrator, go to Admin Area -> Settings, and under Continuous Integration uncheck Disable to prevent CI usage until rake ci:migrate is run (8.0 only).

• (3) If you want to use custom CI settings (e.g. change where builds are stored), please update /etc/gitlab/gitlab.rb (Omnibus) or /home/git/gitlab/config/gitlab.yml (Source).

• (4) Now upgrade GitLab CI to version 8.0. If you are using Omnibus packages, this may have already happened when you upgraded GitLab to 8.0.

• (5) Disable GitLab CI after upgrading to 8.0.

# CI server
# Omnibus
sudo gitlab-ctl stop ci-unicorn
sudo gitlab-ctl stop ci-sidekiq

# Source
sudo service gitlab_ci stop
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec whenever --clear-crontab

Moving data

• (1) Move the database encryption key from your CI server to your GitLab server. The command below will show you what you need to copy-paste to your GitLab server. On Omnibus GitLab servers you will have to add a line to /etc/gitlab/gitlab.rb. On GitLab servers installed from source you will have to replace the contents of /home/git/gitlab/config/secrets.yml.

# CI server
# Omnibus
sudo gitlab-ci-rake backup:show_secrets

# Source
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rake backup:show_secrets RAILS_ENV=production

• (2) Create your final CI data export. If you are converting from MySQL to PostgreSQL, add MYSQL_TO_POSTGRESQL=1 to the end of the rake command. When the command finishes it will print the path to your data export archive; you will need this file later.

# CI server
# Omnibus
sudo gitlab-ci-rake backup:create

# Source
cd /home/git/gitlab
sudo -u git -H bundle exec rake backup:create RAILS_ENV=production

• (3) Copy your CI data archive to your GitLab server. If you were running GitLab and GitLab CI on the same server you can skip this step. There are many ways to do this, below we use SSH agent forwarding and 'scp', which will be easy and fast for most setups. You can also copy the data archive first from the CI server to your laptop and then from your laptop to the GitLab server.

# Start from your laptop
ssh -A ci_admin@ci_server.example
# Now on the CI server
scp /path/to/12345_gitlab_ci_backup.tar gitlab_admin@gitlab_server.example:~

• (4) Make the CI data archive discoverable for GitLab. We assume below that you store backups in the default path, adjust the command if necessary.

# GitLab server
# Omnibus
sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/

# Source
sudo mv /path/to/12345_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups/

• (5) Import the CI data into GitLab.

# GitLab server
# Omnibus
sudo gitlab-rake ci:migrate

# Source
cd /home/git/gitlab
sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production

• (6) Restart GitLab

# GitLab server
# Omnibus
sudo gitlab-ctl hup unicorn
sudo gitlab-ctl restart sidekiq

# Source
sudo service gitlab reload

Redirecting traffic

If you were running GitLab CI with Omnibus packages and you were using the internal NGINX configuration your CI service should now be available both at ci.example.com (the old address) and gitlab.example.com/ci. You are done!

If you installed GitLab CI from source we now need to configure a redirect in NGINX so that existing CI runners can keep using the old CI server address, and so that existing links to your CI server keep working.

1. Update Nginx configuration

To ensure that your existing CI runners are able to communicate with the migrated installation, and that existing build triggers still work, you'll need to update your Nginx configuration to redirect requests for the old locations to the new ones.

Edit /etc/nginx/sites-available/gitlab_ci and paste:

# GITLAB CI
server {
  listen 80 default_server;         # e.g., listen 192.168.1.1:80;
  server_name YOUR_CI_SERVER_FQDN;  # e.g., server_name source.example.com;

  access_log  /var/log/nginx/gitlab_ci_access.log;
  error_log   /var/log/nginx/gitlab_ci_error.log;

  # expose API to fix runners
  location /api {
    proxy_read_timeout    300;
    proxy_connect_timeout 300;
    proxy_redirect        off;
    proxy_set_header      X-Real-IP $remote_addr;

    # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN
    resolver 8.8.8.8 8.8.4.4;
    proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
  }

  # redirect all other CI requests
  location / {
    return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
  }

  # adjust this to match the largest build log your runners might submit,
  # set to 0 to disable limit
  client_max_body_size 10m;
}

Make sure you substitute these placeholder values with your real ones:

1. YOUR_CI_SERVER_FQDN: The existing public-facing address of your GitLab CI install (e.g., ci.gitlab.com).
2. YOUR_GITLAB_SERVER_FQDN: The current public-facing address of your GitLab CE (or EE) install (e.g., gitlab.com).

Make sure not to remove the /ci$request_uri part. This is required to properly forward the requests.

You should also make sure that you can:

1. curl https://YOUR_GITLAB_SERVER_FQDN/ from your previous GitLab CI server.
2. curl https://YOUR_CI_SERVER_FQDN/ from your GitLab CE (or EE) server.

2. Check Nginx configuration

sudo nginx -t

3. Restart Nginx

sudo /etc/init.d/nginx restart

Restore from backup

If something went wrong and you need to restore a backup, consult the Backup restoration guide.