From 67b38c8d715176271076e054d31c14ecc3254bc9 Mon Sep 17 00:00:00 2001 From: Jacob Vosmaer Date: Tue, 22 Sep 2015 13:33:02 +0200 Subject: [PATCH 01/11] Update/improve CI migration docs for 8.0.1 --- doc/migrate_ci_to_ce/README.md | 378 +++++++++++++++------------------ 1 file changed, 168 insertions(+), 210 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 1f050e2def2..c2df33a22e0 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -7,230 +7,198 @@ into the CE and EE applications. This guide will detail the process of migrating your CI installation and data into your GitLab CE or EE installation. -### Before we begin +We recommend that you read through the entire migration process in this +document before beginning. -**You need to have a working installation of GitLab CI version 8.0 to perform -this migration. The older versions are not supported and will most likely break -this migration procedure.** +### Overview -This migration cannot be performed online and takes a significant amount of -time. Make sure to plan ahead. +In this document we assume you have a GitLab server and a GitLab CI server. It +does not matter if these are the same machine. -If you are running a version of GitLab CI prior to 8.0 please follow the -appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/tree/master/doc/update/) -before proceeding. +The migration consists of three parts: updating GitLab and GitLab CI, moving +data, and redirecting traffic. -The migration is divided into four parts and covers both manual and Omnibus -installations: +Please note that CI builds triggered on your GitLab server in the time between +updating to 8.0 and finishing the migration will be lost. -1. [GitLab CI](#part-i-gitlab-ci) -1. [Gitlab CE (or EE)](#part-ii-gitlab-ce-or-ee) -1. [Nginx configuration](#part-iii-nginx-configuration) -1. [Finishing Up](#part-iv-finishing-up) +### Before upgrading -### Part I: GitLab CI +- (1) Make sure that the backup script on both servers can connect to the database. -#### 1. Stop GitLab CI +``` +# CI server +# Omnibus +sudo gitlab-ci-rake backup:create - # Manual installation - sudo service gitlab_ci stop - - # Omnibus installation - sudo gitlab-ctl stop ci-unicorn ci-sidekiq - -#### 2. Create a backup - -The migration procedure modifies the structure of the CI database. If something -goes wrong, you will not be able to revert to a previous version without a -backup. - -If your GitLab CI installation uses **MySQL** and your GitLab CE (or EE) -installation uses **PostgreSQL** you'll need to convert the CI database by -setting a `MYSQL_TO_POSTGRESQL` flag. - -If you use the Omnibus package you most likely use **PostgreSQL** on both GitLab -CE (or EE) and CI. - -You can check which database each install is using by viewing their -database configuration files: - - cat /home/gitlab_ci/gitlab-ci/config/database.yml - cat /home/git/gitlab/config/database.yml - -- If both applications use the same database `adapter`, create the backup with - this command: - - # Manual installation - cd /home/gitlab_ci/gitlab-ci - sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production - - # Omnibus installation - sudo gitlab-ci-rake backup:create - -- If CI uses MySQL, and CE (or EE) uses PostgreSQL, create the backup with this - command (note the `MYSQL_TO_POSTGRESQL` flag): - - # Manual installation - cd /home/gitlab_ci/gitlab-ci - sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1 - - # Omnibus installation - sudo gitlab-ci-rake backup:create MYSQL_TO_POSTGRESQL=1 - -#### 3. Remove cronjob - -**Note:** This step is only required for **manual installations**. Omnibus users -can [skip to the next step](#part-ii-gitlab-ce-or-ee). - - # Manual installation - cd /home/gitlab_ci/gitlab-ci - sudo -u gitlab_ci -H bundle exec whenever --clear-crontab - -### Part II: GitLab CE (or EE) - -#### 1. Ensure GitLab is updated - -Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow -the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md) -before proceeding. - -If you use the Omnibus packages simply run `apt-get upgrade` to install the -latest version. - -#### 2. Prevent CI usage during the migration process - -As an administrator, go to **Admin Area** -> **Settings**, and under **Continuous -Integration** uncheck **Disable to prevent CI usage until rake ci:migrate is run -(8.0 only)**. - -This will disable the CI integration and prevent users from creating CI projects -until the migration process is completed. - -#### 3. Stop GitLab - -Before you can migrate data you need to stop the GitLab service first: - - # Manual installation - sudo service gitlab stop - - # Omnibus installation - sudo gitlab-ctl stop unicorn sidekiq - -#### 4. Create a backup - -This migration poses a **significant risk** of breaking your GitLab -installation. Create a backup before proceeding: - - # Manual installation - cd /home/git/gitlab - sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production - - # Omnibus installation - sudo gitlab-rake gitlab:backup:create - -It's possible to speed up backup creation by skipping repositories and uploads: - - # Manual installation - cd /home/git/gitlab - sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads - - # Omnibus installation - sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads - -#### 5. Copy secret tokens from CI - -The `secrets.yml` file stores encryption keys for secure variables. - -- **Manual installations** need to copy the contents of GitLab CI's - `config/secrets.yml` file to the same file in GitLab CE: - - ```bash - # Manual installation - sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml - sudo chown git:git /home/git/gitlab/config/secrets.yml - sudo chown 0600 /home/git/gitlab/config/secrets.yml - ``` - -- **Omnibus installations** where GitLab CI and CE (or EE) are on the same - server don't need to do anything further, because the secrets are stored in - `/etc/gitlab/gitlab-secrets.json`. - -- **Omnibus installations** where GitLab CI is on a different server than CE (or - EE) will need to: - 1. On the CI server, copy the `db_key_base` value from - `/etc/gitlab/gitlab-secrets.json` - 1. On the CE (or EE) server, add `gitlab_ci['db_key_base'] = - "VALUE_FROM_ABOVE"` to the `/etc/gitlab/gitlab.rb` file and run `sudo - gitlab-ctl reconfigure` - -#### 6. New configuration options for `gitlab.yml` - -**Note:** This step is only required for **manual installations**. Omnibus users -can [skip to the next step](#7-copy-backup-from-gitlab-ci). - -There are new configuration options available for `gitlab.yml`. View them with -the command below and apply them manually to your current `gitlab.yml`: - - git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example - -The new options include configuration settings for GitLab CI. - -#### 7. Copy backup from GitLab CI - -```bash -# Manual installation -sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups -sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar - -# Omnibus installation -sudo cp -v /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar /var/opt/gitlab/backups/ -sudo chown git:git /var/opt/gitlab/backups/*_gitlab_ci_backup.tar +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production ``` -If moving across the servers you can use `scp`. -However, this requires you to provide an authorized key or password to login to -the GitLab CE (or EE) server from the CI server. You can try to use ssh-agent -from your local machine to have that: login to your GitLab CI server using -`ssh -A`. +``` +# GitLab server +# Omnibus +sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads -```bash -# Manual installation -scp /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/home/git/gitlab/tmp/backup - -# Omnibus installation -scp /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/var/opt/gitlab/backups/ +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads ``` -#### 8. Import GitLab CI backup +If this fails you need to fix it before upgrading to 8.0. Also see +https://about.gitlab.com/getting-help/ -Now you'll import the GitLab CI database dump that you created earlier into the -GitLab CE or EE database: +- (2) Check what databases you use on your GitLab server and your CI server. + Look for the 'adapter:' line. If your CI server and your GitLab server use +the same database adapter no special care is needed. If your CI server uses +MySQL and your GitLab server uses PostgreSQL you need to pass a special option +during the 'Moving data' part. **If your CI server uses PostgreSQL and your +GitLab server uses MySQL you cannot migrate your CI data to GitLab 8.0.*** - # Manual installation - sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production +- (3) Decide where to store CI build traces on GitLab server. GitLab CI uses + files on disk to store CI build traces. The default path for these build +traces is `/var/opt/gitlab/gitlab-ci/build` (Omnibus) or +`/home/git/gitlab/builds` (Source). If you are storing your repository data in +a special location, or if you are using NFS, you should make sure that you +store build traces on the same storage as your Git repositories. - # Omnibus installation - sudo gitlab-rake ci:migrate +``` +# CI server +# Omnibus +sudo gitlab-ci-rake env:info -This task will take some time. +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake env:info RAILS_ENV=production +``` -This migration task automatically re-enables the CI setting that you -[disabled earlier](#2-prevent-ci-usage-during-the-migration-process). +``` +# GitLab server +# Omnibus +sudo gitlab-rake gitlab:env:info -#### 9. Start GitLab +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production +``` -You can start GitLab CE (or EE) now and see if everything is working: +### Upgrading - # Manual installation - sudo service gitlab start +From this point on, GitLab CI will be unavailable for your end users. - # Omnibus installation - sudo gitlab-ctl restart unicorn sidekiq +- (1) First upgrade your GitLab server to version 8.0: +https://about.gitlab.com/update/ +- (2) After you update, go to the admin panel and temporarily disable CI. As + an administrator, go to **Admin Area** -> **Settings**, and under +**Continuous Integration** uncheck **Disable to prevent CI usage until rake +ci:migrate is run (8.0 only)**. +- (3) If you want to use custom CI settings (e.g. change where builds are + stored), please update `/etc/gitlab/gitlab.rb` (Omnibus) or +`/home/git/gitlab/config/gitlab.yml` (Source). +- (4) Now upgrade GitLab CI to version 8.0. If you are using Omnibus packages, + this may have already happened when you upgraded GitLab to 8.0. -### Part III: Nginx configuration +- (5) Disable GitLab CI after upgrading to 8.0. -This section is only required for **manual installations**. Omnibus users can -[skip to the final step](#part-iv-finishing-up). +``` +# CI server +# Omnibus +sudo gitlab-ctl stop ci-unicorn +sudo gitlab-ctl stop ci-sidekiq + +# Source +sudo service gitlab_ci stop +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec whenever --clear-crontab +``` + +### Moving data + +- (1) Move the database encryption key from your CI server to your GitLab server. + +``` +# CI server +# Omnibus +sudo gitlab-ci-rake backup:show_secrets + +# Source +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec rake backup:show_secrets RAILS_ENV=production +``` + +- (2) Create your final CI data export. If you are converting from MySQL to + PostgreSQL, add ` MYSQL_TO_POSTGRESQL=1` to the end of the rake command. When +the command finishes it will print the path to your data export archive; you +will need this file later. + +``` +# CI server +# Omnibus +sudo gitlab-ci-rake backup:create + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake backup:create RAILS_ENV=production +``` + +- (3) Copy your CI data archive to your GitLab server. If you were running + GitLab and GitLab CI on the same server you can skip this step. There are +many ways to do this, below we use SSH agent forwarding and 'scp', which will +be easy and fast for most setups. You can also copy the data archive first from +the CI server to your laptop and then from your laptop to the GitLab server. + +``` +# Start from your laptop +ssh -A ci_admin@ci_server.example +# Now on the CI server +scp /path/to/12345_gitlab_ci_backup.tar gitlab_admin@gitlab_server.example:~ +``` + +- (4) Make the CI data archive discoverable for GitLab. We assume below that + you store backups in the default path, adjust the command if necessary. + +``` +# GitLab server +# Omnibus +sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/ + +# Source +sudo mv /path/to/12345_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups/ +``` + +- (5) Import the CI data into GitLab. + +``` +# GitLab server +# Omnibus +sudo gitlab-rake ci:migrate + +# Source +cd /home/git/gitlab +sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production +``` + +- (6) Restart GitLab + +``` +# GitLab server +# Omnibus +sudo gitlab-ctl hup unicorn +sudo gitlab-ctl restart sidekiq + +# Source +sudo service gitlab reload +``` + +### Redirecting traffic + +If you were running GitLab CI with Omnibus packages and you were using the +internal NGINX configuration your CI service should now be available both at +`ci.example.com` (the old address) and `gitlab.example.com/ci`. You are done! + +If you installed GitLab CI from source we now need to configure a redirect in +NGINX so that existing CI runners can keep using the old CI server address, and +so that existing links to your CI server keep working. #### 1. Update Nginx configuration @@ -296,16 +264,6 @@ You should also make sure that you can: sudo /etc/init.d/nginx restart -### Part IV: Finishing Up - -If everything went well you should be able to access your migrated CI install by -visiting `https://gitlab.example.com/ci/`. If you visit the old GitLab CI -address, you should be redirected to the new one. - -**Enjoy!** - -### Troubleshooting - #### Restore from backup If something went wrong and you need to restore a backup, consult the [Backup From c3805c1830e203c6782f06502d2eebf05e9f5745 Mon Sep 17 00:00:00 2001 From: Jacob Vosmaer Date: Tue, 22 Sep 2015 13:40:44 +0200 Subject: [PATCH 02/11] More text outside code blocks --- doc/migrate_ci_to_ce/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index c2df33a22e0..f74cbee9d07 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -35,6 +35,8 @@ cd /home/gitlab_ci/gitlab-ci sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production ``` +Also check on your GitLab server. + ``` # GitLab server # Omnibus From 1afaf5f7fb3074bfc698dca6d598ae3e0d73b21d Mon Sep 17 00:00:00 2001 From: Jacob Vosmaer Date: Tue, 22 Sep 2015 13:42:47 +0200 Subject: [PATCH 03/11] Fix ordering of paragraphs and code blocks --- doc/migrate_ci_to_ce/README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index f74cbee9d07..83fd447ff07 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -55,14 +55,7 @@ https://about.gitlab.com/getting-help/ the same database adapter no special care is needed. If your CI server uses MySQL and your GitLab server uses PostgreSQL you need to pass a special option during the 'Moving data' part. **If your CI server uses PostgreSQL and your -GitLab server uses MySQL you cannot migrate your CI data to GitLab 8.0.*** - -- (3) Decide where to store CI build traces on GitLab server. GitLab CI uses - files on disk to store CI build traces. The default path for these build -traces is `/var/opt/gitlab/gitlab-ci/build` (Omnibus) or -`/home/git/gitlab/builds` (Source). If you are storing your repository data in -a special location, or if you are using NFS, you should make sure that you -store build traces on the same storage as your Git repositories. +GitLab server uses MySQL you cannot migrate your CI data to GitLab 8.0.** ``` # CI server @@ -84,6 +77,13 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production ``` +- (3) Decide where to store CI build traces on GitLab server. GitLab CI uses + files on disk to store CI build traces. The default path for these build +traces is `/var/opt/gitlab/gitlab-ci/build` (Omnibus) or +`/home/git/gitlab/builds` (Source). If you are storing your repository data in +a special location, or if you are using NFS, you should make sure that you +store build traces on the same storage as your Git repositories. + ### Upgrading From this point on, GitLab CI will be unavailable for your end users. From 23fa5d7d9e0ccb3663f4c714efa6c48593dbd449 Mon Sep 17 00:00:00 2001 From: Jacob Vosmaer Date: Tue, 22 Sep 2015 13:44:43 +0200 Subject: [PATCH 04/11] Fix typo --- doc/migrate_ci_to_ce/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 83fd447ff07..ab69f0fde7e 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -79,7 +79,7 @@ sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production - (3) Decide where to store CI build traces on GitLab server. GitLab CI uses files on disk to store CI build traces. The default path for these build -traces is `/var/opt/gitlab/gitlab-ci/build` (Omnibus) or +traces is `/var/opt/gitlab/gitlab-ci/builds` (Omnibus) or `/home/git/gitlab/builds` (Source). If you are storing your repository data in a special location, or if you are using NFS, you should make sure that you store build traces on the same storage as your Git repositories. From 29bb3b52156b41cca9ef6f6f729d66410f47f8ec Mon Sep 17 00:00:00 2001 From: Jacob Vosmaer Date: Tue, 22 Sep 2015 13:52:53 +0200 Subject: [PATCH 05/11] More explanation about moving DB secrets --- doc/migrate_ci_to_ce/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index ab69f0fde7e..1cf23649f6f 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -116,7 +116,11 @@ sudo -u gitlab_ci -H bundle exec whenever --clear-crontab ### Moving data -- (1) Move the database encryption key from your CI server to your GitLab server. +- (1) Move the database encryption key from your CI server to your GitLab + server. The command below will show you what you need to copy-paste to your +GitLab server. On Omnibus GitLab servers you will have to add a line to +`/etc/gitlab/gitlab.rb`. On GitLab servers installed from source you will have +to replace the contents of `/home/git/gitlab/config/secrets.yml`. ``` # CI server From 917c9ea9278d3f3d3ae925351911e8ecc5c55a1f Mon Sep 17 00:00:00 2001 From: Jacob Vosmaer Date: Tue, 22 Sep 2015 13:59:22 +0200 Subject: [PATCH 06/11] Migration only works 8.0 -> 8.0 --- doc/migrate_ci_to_ce/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 1cf23649f6f..4070d15808c 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -5,7 +5,9 @@ Edition (EE), GitLab CI is no longer its own application, but is instead built into the CE and EE applications. This guide will detail the process of migrating your CI installation and data -into your GitLab CE or EE installation. +into your GitLab CE or EE installation. **You can only migrate CI data from +GitLab CI 8.0 to GitLab 8.0; migrating between other versions (e.g.7.14 to 8.1) +is not possible.** We recommend that you read through the entire migration process in this document before beginning. From e72ba928f98358d090fad68ae3794f494ad971cd Mon Sep 17 00:00:00 2001 From: Jacob Vosmaer Date: Tue, 22 Sep 2015 13:59:40 +0200 Subject: [PATCH 07/11] Downtime expectations --- doc/migrate_ci_to_ce/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 4070d15808c..838832d5cd4 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -21,7 +21,10 @@ The migration consists of three parts: updating GitLab and GitLab CI, moving data, and redirecting traffic. Please note that CI builds triggered on your GitLab server in the time between -updating to 8.0 and finishing the migration will be lost. +updating to 8.0 and finishing the migration will be lost. Your GitLab server +can be online for most of the procedure; the only GitLab downtime (if any) is +during the upgrade to 8.0. Your CI service will be offline from the moment you +upgrade to 8.0 until you finish the migration procedure. ### Before upgrading From f0f5be82b05116a1f916ec7a407bf3ea0a195f22 Mon Sep 17 00:00:00 2001 From: Jacob Vosmaer Date: Tue, 22 Sep 2015 14:02:36 +0200 Subject: [PATCH 08/11] Add global numbering --- doc/migrate_ci_to_ce/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 838832d5cd4..6ac79d36884 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -89,7 +89,7 @@ traces is `/var/opt/gitlab/gitlab-ci/builds` (Omnibus) or a special location, or if you are using NFS, you should make sure that you store build traces on the same storage as your Git repositories. -### Upgrading +### I. Upgrading From this point on, GitLab CI will be unavailable for your end users. @@ -119,7 +119,7 @@ cd /home/gitlab_ci/gitlab-ci sudo -u gitlab_ci -H bundle exec whenever --clear-crontab ``` -### Moving data +### II. Moving data - (1) Move the database encryption key from your CI server to your GitLab server. The command below will show you what you need to copy-paste to your @@ -201,7 +201,7 @@ sudo gitlab-ctl restart sidekiq sudo service gitlab reload ``` -### Redirecting traffic +### III. Redirecting traffic If you were running GitLab CI with Omnibus packages and you were using the internal NGINX configuration your CI service should now be available both at From 82d95c461271237eb62eb55846ade3145aa1a9c0 Mon Sep 17 00:00:00 2001 From: Jacob Vosmaer Date: Tue, 22 Sep 2015 14:49:39 +0200 Subject: [PATCH 09/11] Use headings to mark steps --- doc/migrate_ci_to_ce/README.md | 70 +++++++++++++++++++++++++--------- 1 file changed, 51 insertions(+), 19 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 6ac79d36884..534b1fdb2db 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -28,7 +28,9 @@ upgrade to 8.0 until you finish the migration procedure. ### Before upgrading -- (1) Make sure that the backup script on both servers can connect to the database. +#### 1. Verify that backups work + +Make sure that the backup script on both servers can connect to the database. ``` # CI server @@ -55,7 +57,9 @@ sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=r If this fails you need to fix it before upgrading to 8.0. Also see https://about.gitlab.com/getting-help/ -- (2) Check what databases you use on your GitLab server and your CI server. +#### 2. Check source and target database types + +Check what databases you use on your GitLab server and your CI server. Look for the 'adapter:' line. If your CI server and your GitLab server use the same database adapter no special care is needed. If your CI server uses MySQL and your GitLab server uses PostgreSQL you need to pass a special option @@ -82,7 +86,9 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production ``` -- (3) Decide where to store CI build traces on GitLab server. GitLab CI uses +#### 3. Storage planning + +Decide where to store CI build traces on GitLab server. GitLab CI uses files on disk to store CI build traces. The default path for these build traces is `/var/opt/gitlab/gitlab-ci/builds` (Omnibus) or `/home/git/gitlab/builds` (Source). If you are storing your repository data in @@ -93,19 +99,32 @@ store build traces on the same storage as your Git repositories. From this point on, GitLab CI will be unavailable for your end users. -- (1) First upgrade your GitLab server to version 8.0: +#### 1. Upgrade GitLab to 8.0 + +First upgrade your GitLab server to version 8.0: https://about.gitlab.com/update/ -- (2) After you update, go to the admin panel and temporarily disable CI. As + +#### 2. Disable CI on the GitLab server during the migration + +After you update, go to the admin panel and temporarily disable CI. As an administrator, go to **Admin Area** -> **Settings**, and under **Continuous Integration** uncheck **Disable to prevent CI usage until rake ci:migrate is run (8.0 only)**. -- (3) If you want to use custom CI settings (e.g. change where builds are + +#### 3. CI settings are now in GitLab + +If you want to use custom CI settings (e.g. change where builds are stored), please update `/etc/gitlab/gitlab.rb` (Omnibus) or `/home/git/gitlab/config/gitlab.yml` (Source). -- (4) Now upgrade GitLab CI to version 8.0. If you are using Omnibus packages, + +#### 4. Upgrade GitLab CI to 8.0 + +Now upgrade GitLab CI to version 8.0. If you are using Omnibus packages, this may have already happened when you upgraded GitLab to 8.0. -- (5) Disable GitLab CI after upgrading to 8.0. +#### 5. Disable GitLab CI on the CI server + +Disable GitLab CI after upgrading to 8.0. ``` # CI server @@ -121,7 +140,9 @@ sudo -u gitlab_ci -H bundle exec whenever --clear-crontab ### II. Moving data -- (1) Move the database encryption key from your CI server to your GitLab +#### 1. Database encryption key + +Move the database encryption key from your CI server to your GitLab server. The command below will show you what you need to copy-paste to your GitLab server. On Omnibus GitLab servers you will have to add a line to `/etc/gitlab/gitlab.rb`. On GitLab servers installed from source you will have @@ -137,7 +158,9 @@ cd /home/gitlab_ci/gitlab-ci sudo -u gitlab_ci -H bundle exec rake backup:show_secrets RAILS_ENV=production ``` -- (2) Create your final CI data export. If you are converting from MySQL to +#### 2. SQL data and build traces + +Create your final CI data export. If you are converting from MySQL to PostgreSQL, add ` MYSQL_TO_POSTGRESQL=1` to the end of the rake command. When the command finishes it will print the path to your data export archive; you will need this file later. @@ -152,11 +175,15 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake backup:create RAILS_ENV=production ``` -- (3) Copy your CI data archive to your GitLab server. If you were running - GitLab and GitLab CI on the same server you can skip this step. There are -many ways to do this, below we use SSH agent forwarding and 'scp', which will -be easy and fast for most setups. You can also copy the data archive first from -the CI server to your laptop and then from your laptop to the GitLab server. +#### 3. Copy data to the GitLab server + +If you were running GitLab and GitLab CI on the same server you can skip this +step. + +Copy your CI data archive to your GitLab server. There are many ways to do +this, below we use SSH agent forwarding and 'scp', which will be easy and fast +for most setups. You can also copy the data archive first from the CI server to +your laptop and then from your laptop to the GitLab server. ``` # Start from your laptop @@ -165,8 +192,10 @@ ssh -A ci_admin@ci_server.example scp /path/to/12345_gitlab_ci_backup.tar gitlab_admin@gitlab_server.example:~ ``` -- (4) Make the CI data archive discoverable for GitLab. We assume below that - you store backups in the default path, adjust the command if necessary. +#### 4. Move data to the GitLab backups folder + +Make the CI data archive discoverable for GitLab. We assume below that you +store backups in the default path, adjust the command if necessary. ``` # GitLab server @@ -177,7 +206,10 @@ sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/ sudo mv /path/to/12345_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups/ ``` -- (5) Import the CI data into GitLab. +#### 5. Import the CI data into GitLab. + +This step will delete any existing CI data on your GitLab server. There should +be no CI data yet because you turned CI on the GitLab server off earlier. ``` # GitLab server @@ -189,7 +221,7 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production ``` -- (6) Restart GitLab +#### 6. Restart GitLab ``` # GitLab server From 3e926c6eb030eafb10c66c4aade66237e1a41d7a Mon Sep 17 00:00:00 2001 From: Jacob Vosmaer Date: Tue, 22 Sep 2015 14:54:26 +0200 Subject: [PATCH 10/11] Emphasize when Omnibus installs are done --- doc/migrate_ci_to_ce/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 534b1fdb2db..31b572ee8ba 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -237,7 +237,7 @@ sudo service gitlab reload If you were running GitLab CI with Omnibus packages and you were using the internal NGINX configuration your CI service should now be available both at -`ci.example.com` (the old address) and `gitlab.example.com/ci`. You are done! +`ci.example.com` (the old address) and `gitlab.example.com/ci`. **You are done!** If you installed GitLab CI from source we now need to configure a redirect in NGINX so that existing CI runners can keep using the old CI server address, and From d2ab95d1f3bea31e83241ac57ba934eb93c4eb14 Mon Sep 17 00:00:00 2001 From: Jacob Vosmaer Date: Tue, 22 Sep 2015 15:26:40 +0200 Subject: [PATCH 11/11] Make code block comments more verbose --- doc/migrate_ci_to_ce/README.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 31b572ee8ba..208aab42b56 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -33,7 +33,7 @@ upgrade to 8.0 until you finish the migration procedure. Make sure that the backup script on both servers can connect to the database. ``` -# CI server +# On your CI server: # Omnibus sudo gitlab-ci-rake backup:create @@ -45,7 +45,7 @@ sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production Also check on your GitLab server. ``` -# GitLab server +# On your GitLab server: # Omnibus sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads @@ -67,7 +67,7 @@ during the 'Moving data' part. **If your CI server uses PostgreSQL and your GitLab server uses MySQL you cannot migrate your CI data to GitLab 8.0.** ``` -# CI server +# On your CI server: # Omnibus sudo gitlab-ci-rake env:info @@ -77,7 +77,7 @@ sudo -u gitlab_ci -H bundle exec rake env:info RAILS_ENV=production ``` ``` -# GitLab server +# On your GitLab server: # Omnibus sudo gitlab-rake gitlab:env:info @@ -127,7 +127,7 @@ Now upgrade GitLab CI to version 8.0. If you are using Omnibus packages, Disable GitLab CI after upgrading to 8.0. ``` -# CI server +# On your CI server: # Omnibus sudo gitlab-ctl stop ci-unicorn sudo gitlab-ctl stop ci-sidekiq @@ -149,7 +149,7 @@ GitLab server. On Omnibus GitLab servers you will have to add a line to to replace the contents of `/home/git/gitlab/config/secrets.yml`. ``` -# CI server +# On your CI server: # Omnibus sudo gitlab-ci-rake backup:show_secrets @@ -166,7 +166,7 @@ the command finishes it will print the path to your data export archive; you will need this file later. ``` -# CI server +# On your CI server: # Omnibus sudo gitlab-ci-rake backup:create @@ -198,7 +198,7 @@ Make the CI data archive discoverable for GitLab. We assume below that you store backups in the default path, adjust the command if necessary. ``` -# GitLab server +# On your GitLab server: # Omnibus sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/ @@ -212,7 +212,7 @@ This step will delete any existing CI data on your GitLab server. There should be no CI data yet because you turned CI on the GitLab server off earlier. ``` -# GitLab server +# On your GitLab server: # Omnibus sudo gitlab-rake ci:migrate @@ -224,7 +224,7 @@ sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production #### 6. Restart GitLab ``` -# GitLab server +# On your GitLab server: # Omnibus sudo gitlab-ctl hup unicorn sudo gitlab-ctl restart sidekiq