diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 1f050e2def2..c2df33a22e0 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -7,230 +7,198 @@ into the CE and EE applications. This guide will detail the process of migrating your CI installation and data into your GitLab CE or EE installation. -### Before we begin +We recommend that you read through the entire migration process in this +document before beginning. -**You need to have a working installation of GitLab CI version 8.0 to perform -this migration. The older versions are not supported and will most likely break -this migration procedure.** +### Overview -This migration cannot be performed online and takes a significant amount of -time. Make sure to plan ahead. +In this document we assume you have a GitLab server and a GitLab CI server. It +does not matter if these are the same machine. -If you are running a version of GitLab CI prior to 8.0 please follow the -appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/tree/master/doc/update/) -before proceeding. +The migration consists of three parts: updating GitLab and GitLab CI, moving +data, and redirecting traffic. -The migration is divided into four parts and covers both manual and Omnibus -installations: +Please note that CI builds triggered on your GitLab server in the time between +updating to 8.0 and finishing the migration will be lost. -1. [GitLab CI](#part-i-gitlab-ci) -1. [Gitlab CE (or EE)](#part-ii-gitlab-ce-or-ee) -1. [Nginx configuration](#part-iii-nginx-configuration) -1. [Finishing Up](#part-iv-finishing-up) +### Before upgrading -### Part I: GitLab CI +- (1) Make sure that the backup script on both servers can connect to the database. -#### 1. Stop GitLab CI +``` +# CI server +# Omnibus +sudo gitlab-ci-rake backup:create - # Manual installation - sudo service gitlab_ci stop - - # Omnibus installation - sudo gitlab-ctl stop ci-unicorn ci-sidekiq - -#### 2. Create a backup - -The migration procedure modifies the structure of the CI database. If something -goes wrong, you will not be able to revert to a previous version without a -backup. - -If your GitLab CI installation uses **MySQL** and your GitLab CE (or EE) -installation uses **PostgreSQL** you'll need to convert the CI database by -setting a `MYSQL_TO_POSTGRESQL` flag. - -If you use the Omnibus package you most likely use **PostgreSQL** on both GitLab -CE (or EE) and CI. - -You can check which database each install is using by viewing their -database configuration files: - - cat /home/gitlab_ci/gitlab-ci/config/database.yml - cat /home/git/gitlab/config/database.yml - -- If both applications use the same database `adapter`, create the backup with - this command: - - # Manual installation - cd /home/gitlab_ci/gitlab-ci - sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production - - # Omnibus installation - sudo gitlab-ci-rake backup:create - -- If CI uses MySQL, and CE (or EE) uses PostgreSQL, create the backup with this - command (note the `MYSQL_TO_POSTGRESQL` flag): - - # Manual installation - cd /home/gitlab_ci/gitlab-ci - sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1 - - # Omnibus installation - sudo gitlab-ci-rake backup:create MYSQL_TO_POSTGRESQL=1 - -#### 3. Remove cronjob - -**Note:** This step is only required for **manual installations**. Omnibus users -can [skip to the next step](#part-ii-gitlab-ce-or-ee). - - # Manual installation - cd /home/gitlab_ci/gitlab-ci - sudo -u gitlab_ci -H bundle exec whenever --clear-crontab - -### Part II: GitLab CE (or EE) - -#### 1. Ensure GitLab is updated - -Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow -the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md) -before proceeding. - -If you use the Omnibus packages simply run `apt-get upgrade` to install the -latest version. - -#### 2. Prevent CI usage during the migration process - -As an administrator, go to **Admin Area** -> **Settings**, and under **Continuous -Integration** uncheck **Disable to prevent CI usage until rake ci:migrate is run -(8.0 only)**. - -This will disable the CI integration and prevent users from creating CI projects -until the migration process is completed. - -#### 3. Stop GitLab - -Before you can migrate data you need to stop the GitLab service first: - - # Manual installation - sudo service gitlab stop - - # Omnibus installation - sudo gitlab-ctl stop unicorn sidekiq - -#### 4. Create a backup - -This migration poses a **significant risk** of breaking your GitLab -installation. Create a backup before proceeding: - - # Manual installation - cd /home/git/gitlab - sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production - - # Omnibus installation - sudo gitlab-rake gitlab:backup:create - -It's possible to speed up backup creation by skipping repositories and uploads: - - # Manual installation - cd /home/git/gitlab - sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads - - # Omnibus installation - sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads - -#### 5. Copy secret tokens from CI - -The `secrets.yml` file stores encryption keys for secure variables. - -- **Manual installations** need to copy the contents of GitLab CI's - `config/secrets.yml` file to the same file in GitLab CE: - - ```bash - # Manual installation - sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml - sudo chown git:git /home/git/gitlab/config/secrets.yml - sudo chown 0600 /home/git/gitlab/config/secrets.yml - ``` - -- **Omnibus installations** where GitLab CI and CE (or EE) are on the same - server don't need to do anything further, because the secrets are stored in - `/etc/gitlab/gitlab-secrets.json`. - -- **Omnibus installations** where GitLab CI is on a different server than CE (or - EE) will need to: - 1. On the CI server, copy the `db_key_base` value from - `/etc/gitlab/gitlab-secrets.json` - 1. On the CE (or EE) server, add `gitlab_ci['db_key_base'] = - "VALUE_FROM_ABOVE"` to the `/etc/gitlab/gitlab.rb` file and run `sudo - gitlab-ctl reconfigure` - -#### 6. New configuration options for `gitlab.yml` - -**Note:** This step is only required for **manual installations**. Omnibus users -can [skip to the next step](#7-copy-backup-from-gitlab-ci). - -There are new configuration options available for `gitlab.yml`. View them with -the command below and apply them manually to your current `gitlab.yml`: - - git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example - -The new options include configuration settings for GitLab CI. - -#### 7. Copy backup from GitLab CI - -```bash -# Manual installation -sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups -sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar - -# Omnibus installation -sudo cp -v /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar /var/opt/gitlab/backups/ -sudo chown git:git /var/opt/gitlab/backups/*_gitlab_ci_backup.tar +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production ``` -If moving across the servers you can use `scp`. -However, this requires you to provide an authorized key or password to login to -the GitLab CE (or EE) server from the CI server. You can try to use ssh-agent -from your local machine to have that: login to your GitLab CI server using -`ssh -A`. +``` +# GitLab server +# Omnibus +sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads -```bash -# Manual installation -scp /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/home/git/gitlab/tmp/backup - -# Omnibus installation -scp /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/var/opt/gitlab/backups/ +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads ``` -#### 8. Import GitLab CI backup +If this fails you need to fix it before upgrading to 8.0. Also see +https://about.gitlab.com/getting-help/ -Now you'll import the GitLab CI database dump that you created earlier into the -GitLab CE or EE database: +- (2) Check what databases you use on your GitLab server and your CI server. + Look for the 'adapter:' line. If your CI server and your GitLab server use +the same database adapter no special care is needed. If your CI server uses +MySQL and your GitLab server uses PostgreSQL you need to pass a special option +during the 'Moving data' part. **If your CI server uses PostgreSQL and your +GitLab server uses MySQL you cannot migrate your CI data to GitLab 8.0.*** - # Manual installation - sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production +- (3) Decide where to store CI build traces on GitLab server. GitLab CI uses + files on disk to store CI build traces. The default path for these build +traces is `/var/opt/gitlab/gitlab-ci/build` (Omnibus) or +`/home/git/gitlab/builds` (Source). If you are storing your repository data in +a special location, or if you are using NFS, you should make sure that you +store build traces on the same storage as your Git repositories. - # Omnibus installation - sudo gitlab-rake ci:migrate +``` +# CI server +# Omnibus +sudo gitlab-ci-rake env:info -This task will take some time. +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake env:info RAILS_ENV=production +``` -This migration task automatically re-enables the CI setting that you -[disabled earlier](#2-prevent-ci-usage-during-the-migration-process). +``` +# GitLab server +# Omnibus +sudo gitlab-rake gitlab:env:info -#### 9. Start GitLab +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production +``` -You can start GitLab CE (or EE) now and see if everything is working: +### Upgrading - # Manual installation - sudo service gitlab start +From this point on, GitLab CI will be unavailable for your end users. - # Omnibus installation - sudo gitlab-ctl restart unicorn sidekiq +- (1) First upgrade your GitLab server to version 8.0: +https://about.gitlab.com/update/ +- (2) After you update, go to the admin panel and temporarily disable CI. As + an administrator, go to **Admin Area** -> **Settings**, and under +**Continuous Integration** uncheck **Disable to prevent CI usage until rake +ci:migrate is run (8.0 only)**. +- (3) If you want to use custom CI settings (e.g. change where builds are + stored), please update `/etc/gitlab/gitlab.rb` (Omnibus) or +`/home/git/gitlab/config/gitlab.yml` (Source). +- (4) Now upgrade GitLab CI to version 8.0. If you are using Omnibus packages, + this may have already happened when you upgraded GitLab to 8.0. -### Part III: Nginx configuration +- (5) Disable GitLab CI after upgrading to 8.0. -This section is only required for **manual installations**. Omnibus users can -[skip to the final step](#part-iv-finishing-up). +``` +# CI server +# Omnibus +sudo gitlab-ctl stop ci-unicorn +sudo gitlab-ctl stop ci-sidekiq + +# Source +sudo service gitlab_ci stop +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec whenever --clear-crontab +``` + +### Moving data + +- (1) Move the database encryption key from your CI server to your GitLab server. + +``` +# CI server +# Omnibus +sudo gitlab-ci-rake backup:show_secrets + +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake backup:show_secrets RAILS_ENV=production +``` + +- (2) Create your final CI data export. If you are converting from MySQL to + PostgreSQL, add ` MYSQL_TO_POSTGRESQL=1` to the end of the rake command. When +the command finishes it will print the path to your data export archive; you +will need this file later. + +``` +# CI server +# Omnibus +sudo gitlab-ci-rake backup:create + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake backup:create RAILS_ENV=production +``` + +- (3) Copy your CI data archive to your GitLab server. If you were running + GitLab and GitLab CI on the same server you can skip this step. There are +many ways to do this, below we use SSH agent forwarding and 'scp', which will +be easy and fast for most setups. You can also copy the data archive first from +the CI server to your laptop and then from your laptop to the GitLab server. + +``` +# Start from your laptop +ssh -A ci_admin@ci_server.example +# Now on the CI server +scp /path/to/12345_gitlab_ci_backup.tar gitlab_admin@gitlab_server.example:~ +``` + +- (4) Make the CI data archive discoverable for GitLab. We assume below that + you store backups in the default path, adjust the command if necessary. + +``` +# GitLab server +# Omnibus +sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/ + +# Source +sudo mv /path/to/12345_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups/ +``` + +- (5) Import the CI data into GitLab. + +``` +# GitLab server +# Omnibus +sudo gitlab-rake ci:migrate + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production +``` + +- (6) Restart GitLab + +``` +# GitLab server +# Omnibus +sudo gitlab-ctl hup unicorn +sudo gitlab-ctl restart sidekiq + +# Source +sudo service gitlab reload +``` + +### Redirecting traffic + +If you were running GitLab CI with Omnibus packages and you were using the +internal NGINX configuration your CI service should now be available both at +`ci.example.com` (the old address) and `gitlab.example.com/ci`. You are done! + +If you installed GitLab CI from source we now need to configure a redirect in +NGINX so that existing CI runners can keep using the old CI server address, and +so that existing links to your CI server keep working. #### 1. Update Nginx configuration @@ -296,16 +264,6 @@ You should also make sure that you can: sudo /etc/init.d/nginx restart -### Part IV: Finishing Up - -If everything went well you should be able to access your migrated CI install by -visiting `https://gitlab.example.com/ci/`. If you visit the old GitLab CI -address, you should be redirected to the new one. - -**Enjoy!** - -### Troubleshooting - #### Restore from backup If something went wrong and you need to restore a backup, consult the [Backup