2020-10-30 18:08:56 +00:00
---
2021-01-11 15:10:32 +00:00
stage: Enablement
group: Database
2020-11-26 06:09:20 +00:00
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-10-30 18:08:56 +00:00
---
2021-04-20 18:09:37 +00:00
# Migrating from MySQL to PostgreSQL **(FREE SELF)**
2014-05-08 11:18:12 +00:00
2018-12-28 07:43:50 +00:00
This guide documents how to take a working GitLab instance that uses MySQL and
migrate it to a PostgreSQL database.
2014-03-17 13:55:56 +00:00
2018-12-28 07:43:50 +00:00
## Requirements
2018-02-07 22:43:08 +00:00
2020-12-04 21:09:29 +00:00
NOTE:
2019-06-19 13:10:07 +00:00
Support for MySQL was removed in GitLab 12.1. This procedure should be performed
**before** installing GitLab 12.1.
2021-02-03 21:09:17 +00:00
[pgLoader ](https://pgloader.io/ ) 3.4.1+ is required, confirm with `pgloader -V` .
2014-05-08 10:10:04 +00:00
2018-12-28 07:43:50 +00:00
You can install it directly from your distribution, for example in
Debian/Ubuntu:
2014-05-08 10:10:04 +00:00
2018-12-28 07:43:50 +00:00
1. Search for the version:
2014-05-08 11:18:22 +00:00
2020-01-30 15:09:15 +00:00
```shell
2018-12-28 07:43:50 +00:00
apt-cache madison pgloader
```
2014-05-08 11:18:22 +00:00
2018-12-28 07:43:50 +00:00
1. If the version is 3.4.1+, install it with:
2020-01-30 15:09:15 +00:00
```shell
2018-12-28 07:43:50 +00:00
sudo apt-get install pgloader
```
If your distribution's version is too old, use PostgreSQL's repository:
2020-01-30 15:09:15 +00:00
```shell
2018-12-28 07:43:50 +00:00
# Add repository
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
# Add key
sudo apt-get install wget ca-certificates
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
# Install package
sudo apt-get update
sudo apt-get install pgloader
```
2019-04-16 19:20:53 +00:00
For other distributions, follow the instructions in PostgreSQL's
2018-12-28 07:43:50 +00:00
[download page ](https://www.postgresql.org/download/ ) to add their repository
and then install `pgloader` .
2020-12-07 18:10:36 +00:00
If you are migrating to a Docker based installation, you must install
2021-02-03 21:09:17 +00:00
pgLoader within the container as it is not included in the container image.
2019-05-08 02:38:13 +00:00
1. Start a shell session in the context of the running container:
2020-03-03 03:08:31 +00:00
```shell
2019-05-08 02:38:13 +00:00
docker exec -it gitlab bash
```
2021-02-03 21:09:17 +00:00
1. Install pgLoader:
2019-07-08 23:14:29 +00:00
2020-03-03 03:08:31 +00:00
```shell
2019-05-08 02:38:13 +00:00
apt-get update
apt-get -y install pgloader
```
2018-12-28 07:43:50 +00:00
## Omnibus GitLab installations
2020-12-07 18:10:36 +00:00
For [Omnibus GitLab packages ](https://about.gitlab.com/install/ ), you first
2018-12-28 07:43:50 +00:00
need to enable the bundled PostgreSQL:
2014-05-19 16:28:45 +00:00
2017-10-06 09:20:52 +00:00
1. Stop GitLab:
2014-05-19 15:51:58 +00:00
2020-01-30 15:09:15 +00:00
```shell
2019-07-31 10:29:10 +00:00
sudo gitlab-ctl stop
```
2014-09-09 21:55:45 +00:00
2017-10-06 09:20:52 +00:00
1. Edit `/etc/gitlab/gitlab.rb` to enable bundled PostgreSQL:
2014-05-08 10:10:04 +00:00
2020-03-03 03:08:31 +00:00
```ruby
2019-07-31 10:29:10 +00:00
postgresql['enable'] = true
```
2014-05-08 10:10:04 +00:00
2020-12-07 18:10:36 +00:00
1. Edit `/etc/gitlab/gitlab.rb` to use the bundled PostgreSQL. Review all of the
settings beginning with `db_` (such as `gitlab_rails['db_adapter']` ). To use
the default values, you can comment all of them out.
2014-05-08 10:10:04 +00:00
2018-12-28 07:43:50 +00:00
1. [Reconfigure GitLab ](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure )
for the changes to take effect.
2020-12-07 18:10:36 +00:00
2017-10-06 16:51:29 +00:00
1. Start Unicorn and PostgreSQL so that we can prepare the schema:
2017-10-06 09:20:52 +00:00
2020-01-30 15:09:15 +00:00
```shell
2019-07-31 10:29:10 +00:00
sudo gitlab-ctl start unicorn
sudo gitlab-ctl start postgresql
```
2017-10-06 09:20:52 +00:00
1. Run the following commands to prepare the schema:
2020-01-30 15:09:15 +00:00
```shell
2019-07-31 10:29:10 +00:00
sudo gitlab-rake db:create db:migrate
```
2017-10-06 09:20:52 +00:00
2017-10-06 16:51:29 +00:00
1. Stop Unicorn to prevent other database access from interfering with the loading of data:
2014-05-08 10:10:04 +00:00
2020-01-30 15:09:15 +00:00
```shell
2019-07-31 10:29:10 +00:00
sudo gitlab-ctl stop unicorn
```
2014-05-08 10:10:04 +00:00
2020-12-07 18:10:36 +00:00
After these steps, you have a fresh PostgreSQL database with up-to-date schema.
2014-05-08 10:10:04 +00:00
2020-12-07 18:10:36 +00:00
Next, use `pgloader` to migrate the data from the old MySQL database to the
2018-12-28 07:43:50 +00:00
new PostgreSQL one:
2014-05-08 10:10:04 +00:00
2017-10-06 09:20:52 +00:00
1. Save the following snippet in a `commands.load` file, and edit with your
2018-12-28 07:43:50 +00:00
MySQL database `username` , `password` and `host` :
2014-05-08 10:10:04 +00:00
2020-03-03 03:08:31 +00:00
```sql
2019-07-31 10:29:10 +00:00
LOAD DATABASE
FROM mysql://username:password@host/gitlabhq_production
INTO postgresql://gitlab-psql@unix://var/opt/gitlab/postgresql:/gitlabhq_production
2014-05-08 10:10:04 +00:00
2019-07-31 10:29:10 +00:00
WITH include no drop, truncate, disable triggers, create no tables,
create no indexes, preserve index names, no foreign keys,
data only
2015-06-23 13:45:24 +00:00
2019-09-09 14:03:51 +00:00
SET MySQL PARAMETERS
net_read_timeout = '90',
net_write_timeout = '180'
2019-07-31 10:29:10 +00:00
ALTER SCHEMA 'gitlabhq_production' RENAME TO 'public'
2014-05-08 10:10:04 +00:00
2019-07-31 10:29:10 +00:00
;
```
2014-05-08 10:10:04 +00:00
2017-10-06 09:20:52 +00:00
1. Start the migration:
2014-05-08 10:10:04 +00:00
2020-01-30 15:09:15 +00:00
```shell
2019-07-31 10:29:10 +00:00
sudo -u gitlab-psql pgloader commands.load
```
2017-10-06 09:20:52 +00:00
2021-01-21 06:09:03 +00:00
1. After the migration finishes, you should see a summary table that looks like
2019-02-22 13:17:10 +00:00
the following:
2017-10-06 16:51:29 +00:00
2020-03-03 03:08:31 +00:00
```plaintext
2019-07-31 10:29:10 +00:00
table name read imported errors total time
----------------------------------------------- --------- --------- --------- --------------
fetch meta data 119 119 0 0.388s
Truncate 119 119 0 1.134s
----------------------------------------------- --------- --------- --------- --------------
public.abuse_reports 0 0 0 0.490s
public.appearances 0 0 0 0.488s
.
.
.
public.web_hook_logs 0 0 0 1.080s
----------------------------------------------- --------- --------- --------- --------------
COPY Threads Completion 4 4 0 2.008s
Reset Sequences 113 113 0 0.304s
Install Comments 0 0 0 0.000s
----------------------------------------------- --------- --------- --------- --------------
Total import time 1894 1894 0 12.497s
```
If there is no output for more than 30 minutes, it's possible `pgloader` encountered an error. See
the [troubleshooting guide ](#troubleshooting ) for more details.
2017-10-06 16:51:29 +00:00
1. Start GitLab:
2017-10-06 09:20:52 +00:00
2020-01-30 15:09:15 +00:00
```shell
2019-07-31 10:29:10 +00:00
sudo gitlab-ctl start
```
2017-10-06 09:20:52 +00:00
2018-12-28 07:43:50 +00:00
You can now verify that everything works as expected by visiting GitLab.
2018-02-07 22:43:08 +00:00
2018-12-28 07:43:50 +00:00
## Source installations
2018-02-07 22:43:08 +00:00
2020-12-07 18:10:36 +00:00
For installations from source that use MySQL, you must first
2018-12-28 07:43:50 +00:00
[install PostgreSQL and create a database ](../install/installation.md#6-database ).
2018-02-07 22:43:08 +00:00
2018-12-28 07:43:50 +00:00
After the database is created, go on with the following steps:
2018-02-07 22:43:08 +00:00
1. Stop GitLab:
2020-01-30 15:09:15 +00:00
```shell
2018-12-28 07:43:50 +00:00
sudo service gitlab stop
```
2018-02-07 22:43:08 +00:00
1. Switch database from MySQL to PostgreSQL
2020-01-30 15:09:15 +00:00
```shell
2018-12-28 07:43:50 +00:00
cd /home/git/gitlab
sudo -u git mv config/database.yml config/database.yml.bak
sudo -u git cp config/database.yml.postgresql config/database.yml
sudo -u git -H chmod o-rwx config/database.yml
```
2019-10-02 12:06:04 +00:00
1. Install Gems related to PostgreSQL
2018-07-20 14:00:53 +00:00
2020-01-30 15:09:15 +00:00
```shell
2018-12-28 07:43:50 +00:00
sudo -u git -H rm .bundle/config
sudo -u git -H bundle install --deployment --without development test mysql aws kerberos
```
2018-02-07 22:43:08 +00:00
1. Run the following commands to prepare the schema:
2020-01-30 15:09:15 +00:00
```shell
2018-12-28 07:43:50 +00:00
sudo -u git -H bundle exec rake db:create db:migrate RAILS_ENV=production
```
2018-02-07 22:43:08 +00:00
2020-12-07 18:10:36 +00:00
After these steps, you have a fresh PostgreSQL database with up-to-date schema.
2018-02-07 22:43:08 +00:00
2020-12-07 18:10:36 +00:00
Next, use `pgloader` to migrate the data from the old MySQL database to the
2018-12-28 07:43:50 +00:00
new PostgreSQL one:
2018-02-07 22:43:08 +00:00
1. Save the following snippet in a `commands.load` file, and edit with your
MySQL `username` , `password` and `host` :
2020-03-03 03:08:31 +00:00
```sql
2018-12-28 07:43:50 +00:00
LOAD DATABASE
FROM mysql://username:password@host/gitlabhq_production
INTO postgresql://postgres@unix://var/run/postgresql:/gitlabhq_production
2018-02-07 22:43:08 +00:00
2018-12-28 07:43:50 +00:00
WITH include no drop, truncate, disable triggers, create no tables,
create no indexes, preserve index names, no foreign keys,
data only
2019-09-09 15:04:25 +00:00
2019-09-09 14:03:51 +00:00
SET MySQL PARAMETERS
net_read_timeout = '90',
2019-09-09 15:04:25 +00:00
net_write_timeout = '180'
2018-02-07 22:43:08 +00:00
2018-12-28 07:43:50 +00:00
ALTER SCHEMA 'gitlabhq_production' RENAME TO 'public'
2018-02-07 22:43:08 +00:00
2018-12-28 07:43:50 +00:00
;
```
2018-02-07 22:43:08 +00:00
1. Start the migration:
2020-01-30 15:09:15 +00:00
```shell
2018-12-28 07:43:50 +00:00
sudo -u postgres pgloader commands.load
```
2018-02-07 22:43:08 +00:00
2021-01-21 06:09:03 +00:00
1. After the migration finishes, you should see a summary table that looks like
2019-02-22 13:17:10 +00:00
the following:
2018-02-07 22:43:08 +00:00
2020-03-03 03:08:31 +00:00
```plaintext
2018-12-28 07:43:50 +00:00
table name read imported errors total time
----------------------------------------------- --------- --------- --------- --------------
fetch meta data 119 119 0 0.388s
Truncate 119 119 0 1.134s
----------------------------------------------- --------- --------- --------- --------------
public.abuse_reports 0 0 0 0.490s
public.appearances 0 0 0 0.488s
.
.
.
public.web_hook_logs 0 0 0 1.080s
----------------------------------------------- --------- --------- --------- --------------
COPY Threads Completion 4 4 0 2.008s
Reset Sequences 113 113 0 0.304s
Install Comments 0 0 0 0.000s
----------------------------------------------- --------- --------- --------- --------------
Total import time 1894 1894 0 12.497s
```
If there is no output for more than 30 minutes, it's possible `pgloader` encountered an error. See
the [troubleshooting guide ](#troubleshooting ) for more details.
2018-02-07 22:43:08 +00:00
1. Start GitLab:
2020-01-30 15:09:15 +00:00
```shell
2018-12-28 07:43:50 +00:00
sudo service gitlab start
```
You can now verify that everything works as expected by visiting GitLab.
## Troubleshooting
Sometimes, you might encounter some errors during or after the migration.
2018-02-07 22:43:08 +00:00
2018-12-28 07:43:50 +00:00
### Database error permission denied
2018-02-07 22:43:08 +00:00
2018-12-28 07:43:50 +00:00
The PostgreSQL user that you use for the migration MUST have **superuser** privileges.
Otherwise, you may see a similar message to the following:
2018-02-07 22:43:08 +00:00
2020-03-03 03:08:31 +00:00
```plaintext
2018-12-28 07:43:50 +00:00
debugger invoked on a CL-POSTGRES-ERROR:INSUFFICIENT-PRIVILEGE in thread
#< THREAD " lparallel " RUNNING { 10078A3513 } > :
Database error 42501: permission denied: "RI_ConstraintTrigger_a_20937" is a system trigger
QUERY: ALTER TABLE ci_builds DISABLE TRIGGER ALL;
2017-08-23T00:36:56.782000Z ERROR Database error 42501: permission denied: "RI_ConstraintTrigger_c_20864" is a system trigger
QUERY: ALTER TABLE approver_groups DISABLE TRIGGER ALL;
```
### Experiencing 500 errors after the migration
2018-02-07 22:43:08 +00:00
If you experience 500 errors after the migration, try to clear the cache:
2020-01-30 15:09:15 +00:00
```shell
2018-12-28 07:43:50 +00:00
# Omnibus GitLab
sudo gitlab-rake cache:clear
# Installations from source
2018-02-07 22:43:08 +00:00
sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production
```