Reformat and copy edit the CI-to-CE migration guide
This commit is contained in:
parent
0aec0d53b1
commit
567eac0396
|
@ -1,96 +1,105 @@
|
|||
## Migrate GitLab CI to GitLab CE/EE
|
||||
## Migrate GitLab CI to GitLab CE or EE
|
||||
|
||||
## Notice
|
||||
Beginning with version 8.0 of GitLab Community Edition (CE) and Enterprise
|
||||
Edition (EE), GitLab CI is no longer its own application, but is instead built
|
||||
into the CE and EE applications.
|
||||
|
||||
**You need to have working GitLab CI 7.14 to perform migration.
|
||||
The older versions are not supported and will most likely break migration procedure.**
|
||||
This guide will detail the process of migrating your CI installation and data
|
||||
into your GitLab CE or EE installation.
|
||||
|
||||
This migration can't be done online and takes significant amount of time.
|
||||
Make sure to plan it ahead.
|
||||
### Before we begin
|
||||
|
||||
If you are running older version please follow the upgrade guide first:
|
||||
https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/update/7.13-to-7.14.md
|
||||
**You need to have a working installation of GitLab CI version 7.14 to perform
|
||||
this migration. The older versions are not supported and will most likely break
|
||||
this migration procedure.**
|
||||
|
||||
The migration is divided into a two parts:
|
||||
1. **[CI]** You will be making a changes to GitLab CI instance.
|
||||
1. **[CE]** You will be making a changes to GitLab CE/EE instance.
|
||||
This migration cannot be performed online and takes a significant amount of
|
||||
time. Make sure to plan ahead.
|
||||
|
||||
### 1. Stop CI server [CI]
|
||||
If you are running a version of GitLab CI prior to 7.14 please follow the
|
||||
appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/update/).
|
||||
|
||||
The migration is divided into three parts:
|
||||
|
||||
1. [GitLab CI](#part-i-gitlab-ci)
|
||||
1. [Gitlab CE (or EE)](#part-ii-gitlab-ce-or-ee)
|
||||
1. [Finishing Up](#part-iii-finishing-up)
|
||||
|
||||
### Part I: GitLab CI
|
||||
|
||||
#### 1. Stop GitLab CI
|
||||
|
||||
sudo service gitlab_ci stop
|
||||
|
||||
### 2. Backup [CI]
|
||||
#### 2. Create a backup
|
||||
|
||||
**The migration procedure is database breaking.
|
||||
You need to create backup if you still want to access GitLab CI in case of failure.**
|
||||
The migration procedure modifies the structure of the CI database. If something
|
||||
goes wrong, you will not be able to revert to a previous version without a
|
||||
backup:
|
||||
|
||||
```bash
|
||||
cd /home/gitlab_ci/gitlab-ci
|
||||
sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production
|
||||
```
|
||||
|
||||
### 3. Prepare GitLab CI database to migration [CI]
|
||||
#### 3. Rename database tables
|
||||
|
||||
Copy and paste the command in terminal to rename all tables.
|
||||
This also breaks your database structure disallowing you to use it anymore.
|
||||
To prevent naming conflicts with database tables in GitLab CE or EE, we need to
|
||||
rename CI's tables to begin with a `ci_` prefix:
|
||||
|
||||
cat <<EOF | bundle exec rails dbconsole production
|
||||
ALTER TABLE application_settings RENAME TO ci_application_settings;
|
||||
ALTER TABLE builds RENAME TO ci_builds;
|
||||
ALTER TABLE commits RENAME TO ci_commits;
|
||||
ALTER TABLE events RENAME TO ci_events;
|
||||
ALTER TABLE jobs RENAME TO ci_jobs;
|
||||
ALTER TABLE projects RENAME TO ci_projects;
|
||||
ALTER TABLE runner_projects RENAME TO ci_runner_projects;
|
||||
ALTER TABLE runners RENAME TO ci_runners;
|
||||
ALTER TABLE services RENAME TO ci_services;
|
||||
ALTER TABLE tags RENAME TO ci_tags;
|
||||
ALTER TABLE taggings RENAME TO ci_taggings;
|
||||
ALTER TABLE trigger_requests RENAME TO ci_trigger_requests;
|
||||
ALTER TABLE triggers RENAME TO ci_triggers;
|
||||
ALTER TABLE variables RENAME TO ci_variables;
|
||||
ALTER TABLE web_hooks RENAME TO ci_web_hooks;
|
||||
EOF
|
||||
```sh
|
||||
cat <<EOF | bundle exec rails dbconsole production
|
||||
ALTER TABLE application_settings RENAME TO ci_application_settings;
|
||||
ALTER TABLE builds RENAME TO ci_builds;
|
||||
ALTER TABLE commits RENAME TO ci_commits;
|
||||
ALTER TABLE events RENAME TO ci_events;
|
||||
ALTER TABLE jobs RENAME TO ci_jobs;
|
||||
ALTER TABLE projects RENAME TO ci_projects;
|
||||
ALTER TABLE runner_projects RENAME TO ci_runner_projects;
|
||||
ALTER TABLE runners RENAME TO ci_runners;
|
||||
ALTER TABLE services RENAME TO ci_services;
|
||||
ALTER TABLE tags RENAME TO ci_tags;
|
||||
ALTER TABLE taggings RENAME TO ci_taggings;
|
||||
ALTER TABLE trigger_requests RENAME TO ci_trigger_requests;
|
||||
ALTER TABLE triggers RENAME TO ci_triggers;
|
||||
ALTER TABLE variables RENAME TO ci_variables;
|
||||
ALTER TABLE web_hooks RENAME TO ci_web_hooks;
|
||||
EOF
|
||||
```
|
||||
|
||||
### 4. Remove CI cronjob
|
||||
#### 4. Remove cronjob
|
||||
|
||||
```
|
||||
cd /home/gitlab_ci/gitlab-ci
|
||||
sudo -u gitlab_ci -H bundle exec whenever --clear-crontab
|
||||
```
|
||||
|
||||
### 5. Dump GitLab CI database [CI]
|
||||
#### 5. Create a database dump
|
||||
|
||||
First check used database and credentials on GitLab CI and GitLab CE/EE:
|
||||
In this step, you will need to know information about both your CI and CE (or
|
||||
EE) databases, such as the server types, hosts, and ports, and the usernames and
|
||||
passwords.
|
||||
|
||||
1. To check it on GitLab CI:
|
||||
We can obtain the necessary information from the `config/database.yml` files for
|
||||
each installation.
|
||||
|
||||
1. Get the information for the CI database:
|
||||
|
||||
```sh
|
||||
cat /home/gitlab_ci/gitlab-ci/config/database.yml
|
||||
```
|
||||
|
||||
1. To check it on GitLab CE/EE:
|
||||
1. Then for the CE (or EE) database:
|
||||
|
||||
```sh
|
||||
cat /home/git/gitlab/config/database.yml
|
||||
```
|
||||
|
||||
Please first check the database engine used for GitLab CI and GitLab CE/EE.
|
||||
1. The output of each command should look something like this:
|
||||
|
||||
1. If your GitLab CI uses **mysql2** and GitLab CE/EE uses it too.
|
||||
Please follow **Dump MySQL** guide.
|
||||
|
||||
1. If your GitLab CI uses **postgres** and GitLab CE/EE uses **postgres**.
|
||||
Please follow **Dump PostgreSQL** guide.
|
||||
|
||||
1. If your GitLab CI uses **mysql2** and GitLab CE/EE uses **postgres**.
|
||||
Please follow **Dump MySQL and migrate to PostgreSQL** guide.
|
||||
|
||||
**Remember credentials stored for accessing GitLab CI.
|
||||
You will need to put these credentials into commands executed below.**
|
||||
|
||||
$ cat config/database.yml [10:06:55]
|
||||
#
|
||||
# PRODUCTION
|
||||
#
|
||||
```yml
|
||||
production:
|
||||
adapter: postgresql or mysql2
|
||||
adapter: postgresql (or mysql2)
|
||||
encoding: utf8
|
||||
reconnect: false
|
||||
database: GITLAB_CI_DATABASE
|
||||
|
@ -100,26 +109,42 @@ You will need to put these credentials into commands executed below.**
|
|||
host: DB_HOSTNAME
|
||||
port: DB_PORT
|
||||
# socket: /tmp/mysql.sock
|
||||
```
|
||||
|
||||
#### a. Dump MySQL
|
||||
1. Depending on the values for `adapter`, you will have to use different
|
||||
commands to perform the database dump.
|
||||
|
||||
**NOTE:** For any of the commands below, you'll need to substitute the
|
||||
values `IN_UPPERCASE` with the corresponding values from your **CI
|
||||
installation's** `config/database.yml` files above.
|
||||
|
||||
- If both your CI and CE (or EE) installations use **mysql2** as the `adapter`, use
|
||||
`mysqldump`:
|
||||
|
||||
```sh
|
||||
mysqldump --default-character-set=utf8 --complete-insert --no-create-info \
|
||||
--host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p
|
||||
GITLAB_CI_DATABASE \
|
||||
--host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \
|
||||
ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \
|
||||
ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \
|
||||
ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql
|
||||
```
|
||||
|
||||
#### b. Dump PostgreSQL
|
||||
- If both your CI and CE (or EE) installations use **postgresql** as the
|
||||
`adapter`, use `pg_dump`:
|
||||
|
||||
pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql
|
||||
```sh
|
||||
pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT \
|
||||
--data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql
|
||||
```
|
||||
|
||||
#### c. Dump MySQL and migrate to PostgreSQL
|
||||
- If your CI installation uses **mysql2** as the `adapter` and your CE (or
|
||||
EE) installation uses **postgresql**, use `mysqldump` to dump the database
|
||||
and then convert it to PostgreSQL using [mysql-postgresql-converter]:
|
||||
|
||||
```sh
|
||||
# Dump existing MySQL database first
|
||||
mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \
|
||||
--host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p
|
||||
GITLAB_CI_DATABASE \
|
||||
--host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \
|
||||
ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \
|
||||
ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \
|
||||
ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp
|
||||
|
@ -131,87 +156,101 @@ You will need to put these credentials into commands executed below.**
|
|||
|
||||
# Filter to only include INSERT statements
|
||||
grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql
|
||||
```
|
||||
|
||||
### 6. Make sure that your GitLab CE/EE is 8.0 [CE]
|
||||
[mysql-postgresql-converter]: https://github.com/gitlabhq/mysql-postgresql-converter
|
||||
|
||||
Please verify that you use GitLab CE/EE 8.0.
|
||||
If not, please follow the update guide: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md
|
||||
### Part II: GitLab CE (or EE)
|
||||
|
||||
### 7. Stop GitLab CE/EE [CE]
|
||||
#### 1. Ensure GitLab is updated
|
||||
|
||||
Before you can migrate data you need to stop GitLab CE/EE first.
|
||||
Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow
|
||||
the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md).
|
||||
|
||||
#### 2. Stop GitLab
|
||||
|
||||
Before you can migrate data you need to stop the GitLab service first:
|
||||
|
||||
sudo service gitlab stop
|
||||
|
||||
### 8. Backup GitLab CE/EE [CE]
|
||||
#### 3. Create a backup
|
||||
|
||||
This migration poses a **significant risk** of breaking your GitLab CE/EE.
|
||||
**You should create the GitLab CI/EE backup before doing it.**
|
||||
This migration poses a **significant risk** of breaking your GitLab
|
||||
installation. Create a backup before proceeding:
|
||||
|
||||
cd /home/git/gitlab
|
||||
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
|
||||
|
||||
### 9. Copy secret tokens [CE]
|
||||
#### 4. Copy secret tokens from CI
|
||||
|
||||
The `secrets.yml` file stores encryption keys for secure variables.
|
||||
|
||||
You need to copy the content of `config/secrets.yml` to the same file in GitLab CE.
|
||||
You need to copy the contents of GitLab CI's `config/secrets.yml` file to the
|
||||
same file in GitLab CE:
|
||||
|
||||
sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml
|
||||
sudo chown git:git /home/git/gitlab/config/secrets.yml
|
||||
sudo chown 0600 /home/git/gitlab/config/secrets.yml
|
||||
|
||||
### 10. New configuration options for `gitlab.yml` [CE]
|
||||
#### 5. New configuration options for `gitlab.yml`
|
||||
|
||||
There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example).
|
||||
View them with the command below and apply them manually to your current `gitlab.yml`:
|
||||
There are new configuration options available for `gitlab.yml`. View them with
|
||||
the command below and apply them manually to your current `gitlab.yml`:
|
||||
|
||||
```sh
|
||||
git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example
|
||||
```
|
||||
|
||||
The new options include configuration of GitLab CI that are now being part of GitLab CE and EE.
|
||||
The new options include configuration settings for GitLab CI.
|
||||
|
||||
### 11. Copy build logs [CE]
|
||||
#### 6. Copy build logs
|
||||
|
||||
You need to copy the contents of `builds/` to the same directory in GitLab CE/EE.
|
||||
You need to copy the contents of GitLab CI's `builds/` directory to the
|
||||
corresponding directory in GitLab CE or EE:
|
||||
|
||||
sudo rsync -av /home/gitlab_ci/gitlab-ci/builds /home/git/gitlab/builds
|
||||
sudo chown -R git:git /home/git/gitlab/builds
|
||||
|
||||
The build traces are usually quite big so it will take a significant amount of time.
|
||||
The build logs are usually quite big so it may take a significant amount of
|
||||
time.
|
||||
|
||||
### 12. Import GitLab CI database [CE]
|
||||
#### 7. Import GitLab CI database
|
||||
|
||||
The one of the last steps is to import existing GitLab CI database.
|
||||
Now you'll import the GitLab CI database dump that you [created
|
||||
earlier](#create-a-database-dump) into the GitLab CE or EE database:
|
||||
|
||||
sudo mv /home/gitlab_ci/gitlab-ci/gitlab_ci.sql /home/git/gitlab/gitlab_ci.sql
|
||||
sudo chown git:git /home/git/gitlab/gitlab_ci.sql
|
||||
sudo -u git -H bundle exec rake ci:migrate CI_DUMP=/home/git/gitlab/gitlab_ci.sql RAILS_ENV=production
|
||||
|
||||
The task does:
|
||||
This task will:
|
||||
|
||||
1. Delete data from all existing CI tables
|
||||
1. Import database data
|
||||
1. Fix database auto increments
|
||||
1. Import data from database dump
|
||||
1. Fix database auto-increments
|
||||
1. Fix tags assigned to Builds and Runners
|
||||
1. Fix services used by CI
|
||||
|
||||
### 13. Start GitLab [CE]
|
||||
#### 8. Start GitLab
|
||||
|
||||
You can start GitLab CI/EE now and see if everything is working.
|
||||
You can start GitLab CI (or EE) now and see if everything is working:
|
||||
|
||||
sudo service gitlab start
|
||||
|
||||
### 14. Update nginx [CI]
|
||||
### Part III: Finishing Up
|
||||
|
||||
Now get back to GitLab CI and update **Nginx** configuration in order to:
|
||||
1. Have all existing runners able to communicate with a migrated GitLab CI.
|
||||
1. Have GitLab able send build triggers to CI address specified in Project's settings -> Services -> GitLab CI.
|
||||
#### 1. Update Nginx configuration
|
||||
|
||||
You need to edit `/etc/nginx/sites-available/gitlab_ci` and paste:
|
||||
To ensure that your existing CI runners are able to communicate with the
|
||||
migrated installation, and that existing build triggers still work, you'll need
|
||||
to update your Nginx configuration to redirect requests for the old locations to
|
||||
the new ones.
|
||||
|
||||
# GITLAB CI
|
||||
server {
|
||||
Edit `/etc/nginx/sites-available/gitlab_ci` and paste:
|
||||
|
||||
```nginx
|
||||
# GITLAB CI
|
||||
server {
|
||||
listen 80 default_server; # e.g., listen 192.168.1.1:80;
|
||||
server_name YOUR_CI_SERVER_FQDN; # e.g., server_name source.example.com;
|
||||
|
||||
|
@ -250,31 +289,36 @@ You need to edit `/etc/nginx/sites-available/gitlab_ci` and paste:
|
|||
# adjust this to match the largest build log your runners might submit,
|
||||
# set to 0 to disable limit
|
||||
client_max_body_size 10m;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Make sure to fill the blanks to match your setup:
|
||||
1. **YOUR_CI_SERVER_FQDN**: The existing public facing address of GitLab CI, eg. ci.gitlab.com.
|
||||
1. **YOUR_GITLAB_SERVER_FQDN**: The public facing address of GitLab CE/EE, eg. gitlab.com.
|
||||
Make sure you substitute these placeholder values with your real ones:
|
||||
|
||||
**Make sure to not remove the `/ci$request_uri`. This is required to properly forward the requests.**
|
||||
1. `YOUR_CI_SERVER_FQDN`: The existing public-facing address of your GitLab CI
|
||||
install (e.g., `ci.gitlab.com`).
|
||||
1. `YOUR_GITLAB_SERVER_FQDN`: The current public-facing address of your GitLab
|
||||
CE (or EE) install (e.g., `gitlab.com`).
|
||||
|
||||
**Make sure not to remove the `/ci$request_uri` part. This is required to properly forward the requests.**
|
||||
|
||||
You should also make sure that you can:
|
||||
|
||||
You should also make sure that you can do:
|
||||
1. `curl https://YOUR_GITLAB_SERVER_FQDN/` from your previous GitLab CI server.
|
||||
1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE/EE server.
|
||||
1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE (or EE) server.
|
||||
|
||||
## Check your configuration
|
||||
#### 2. Check Nginx configuration
|
||||
|
||||
sudo nginx -t
|
||||
|
||||
## Restart nginx
|
||||
#### 3. Restart Nginx
|
||||
|
||||
sudo /etc/init.d/nginx restart
|
||||
|
||||
### 15. Done!
|
||||
#### 4. Done!
|
||||
|
||||
If everything went OK you should be able to access all your GitLab CI data by pointing your browser to:
|
||||
https://gitlab.example.com/ci/.
|
||||
If everything went well you should be able to access your migrated CI install by
|
||||
visiting `https://gitlab.example.com/ci/`.
|
||||
|
||||
The GitLab CI should also work when using the previous address, redirecting you to the GitLab CE/EE.
|
||||
If you visit the old GitLab CI address, you should be redirected to the new one.
|
||||
|
||||
**Enjoy!**
|
||||
|
|
Loading…
Reference in New Issue