kotovalexarian-likes-gitlab/gitlab-org--gitlab-foss
1
0
Fork 0
Code Releases Activity

Merge branch 'tc-ce-archive-old-geo-update-docs' into 'master'

Browse source 
Separate doc for version specific Geo update steps

Closes gitlab-ee#11881

See merge request gitlab-org/gitlab-ce!32542
This commit is contained in:
Evan Read 2019-09-05 04:02:07 +00:00
commit 448c2761fa
3 changed files with 448 additions and 427 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
doc/administration/geo/disaster_recovery/index.md
View file

@ -315,7 +315,7 @@ section to resolve the error. Otherwise, the secret is lost and you'll need to
[geo-limitations]: ../replication/index.md#current-limitations
[planned-failover]: planned_failover.md
[setup-geo]: ../replication/index.md#setup-instructions
[updating-geo]: ../replication/updating_the_geo_nodes.md#upgrading-to-gitlab-105
[updating-geo]: ../replication/version_specific_updates.md#updating-to-gitlab-105
[sec-tfa]: ../../../security/two_factor_authentication.md#disabling-2fa-for-everyone
[gitlab-org/omnibus-gitlab#3058]: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3058
[gitlab-org/gitlab-ee#4284]: https://gitlab.com/gitlab-org/gitlab-ee/issues/4284

447
doc/administration/geo/replication/updating_the_geo_nodes.md
View file

@ -1,6 +1,26 @@
# Updating the Geo nodes **(PREMIUM ONLY)**
Depending on which version of Geo you are updating to/from, there may be different steps.
Updating Geo nodes involves performing:
1. [Version-specific update steps](#version-specific-update-steps), depending on the
version being updated to or from.
1. [General update steps](#general-update-steps), for all updates.
## Version specific update steps
Depending on which version of Geo you are updating to/from, there may be
different steps.
- [Updating to GitLab 12.1](version_specific_updates.md#updating-to-gitlab-121)
- [Updating to GitLab 10.8](version_specific_updates.md#updating-to-gitlab-108)
- [Updating to GitLab 10.6](version_specific_updates.md#updating-to-gitlab-106)
- [Updating to GitLab 10.5](version_specific_updates.md#updating-to-gitlab-105)
- [Updating to GitLab 10.3](version_specific_updates.md#updating-to-gitlab-103)
- [Updating to GitLab 10.2](version_specific_updates.md#updating-to-gitlab-102)
- [Updating to GitLab 10.1](version_specific_updates.md#updating-to-gitlab-101)
- [Updating to GitLab 10.0](version_specific_updates.md#updating-to-gitlab-100)
- [Updating from GitLab 9.3 or older](version_specific_updates.md#updating-from-gitlab-93-or-older)
- [Updating to GitLab 9.0](version_specific_updates.md#updating-to-gitlab-90)
## General update steps
@ -31,428 +51,3 @@ everything is working correctly:
is received by **secondary** nodes.
If you encounter any issues, please consult the [Geo troubleshooting guide](troubleshooting.md).
## Upgrading to GitLab 12.1
By default, GitLab 12.1 will attempt to automatically upgrade the embedded PostgreSQL server to 10.7 from 9.6. Please see [the omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) for the recommended procedure.
This can be temporarily disabled by running the following before ugprading:
```sh
sudo touch /etc/gitlab/disable-postgresql-upgrade
```
## Upgrading to GitLab 10.8
Before 10.8, broadcast messages would not propagate without flushing the cache on the **secondary** nodes. This has been fixed in 10.8, but requires one last cache flush on each **secondary** node:
```sh
sudo gitlab-rake cache:clear
```
## Upgrading to GitLab 10.6
In 10.4, we started to recommend that you define a password for database user (`gitlab`).
We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize
the Geo Tracking Database. We are also improving security by disabling the use of **trust**
authentication method.
1. **(primary)** Login to your **primary** node and run:
```sh
gitlab-ctl pg-password-md5 gitlab
# Enter password: <your_password_here>
# Confirm password: <your_password_here>
# fca0b89a972d69f00eb3ec98a5838484
```
Copy the generated hash and edit `/etc/gitlab/gitlab.rb`:
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below. If you have a high-availability setup, this
# must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'
```
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
```ruby
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this
```
1. **(primary)** Reconfigure and restart:
```sh
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
```
1. **(secondary)** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`:
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below. If you have a high-availability setup, this
# must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'
# Enable Foreign Data Wrapper
geo_secondary['db_fdw'] = true
# Secondary address in CIDR format, for example '5.6.7.8/32'
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
```
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
```ruby
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this
```
1. **(secondary)** Reconfigure and restart:
```sh
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
```
## Upgrading to GitLab 10.5
For Geo Disaster Recovery to work with minimum downtime, your **secondary** node
should use the same set of secrets as the **primary** node. However, setup instructions
prior to the 10.5 release only synchronized the `db_key_base` secret.
To rectify this error on existing installations, you should **overwrite** the
contents of `/etc/gitlab/gitlab-secrets.json` on each **secondary** node with the
contents of `/etc/gitlab/gitlab-secrets.json` on the **primary** node, then run the
following command on each **secondary** node:
```sh
sudo gitlab-ctl reconfigure
```
If you do not perform this step, you may find that two-factor authentication
[is broken following DR](../disaster_recovery/index.html#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken).
To prevent SSH requests to the newly promoted **primary** node from failing
due to SSH host key mismatch when updating the **primary** node domain's DNS record
you should perform the step to [Manually replicate **primary** SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys) in each
**secondary** node.
## Upgrading to GitLab 10.4
There are no Geo-specific steps to take!
## Upgrading to GitLab 10.3
### Support for SSH repository synchronization removed
In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3,
support is removed entirely. All installations will switch to the HTTP/HTTPS
cloning method instead. Before upgrading, ensure that all your Geo nodes are
configured to use this method and that it works for your installation. In
particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-6-enable-git-access-over-httphttps).
Synchronizing repositories over the public Internet using HTTP is insecure, so
you should ensure that you have HTTPS configured before upgrading. Note that
file synchronization is **also** insecure in these cases!
## Upgrading to GitLab 10.2
### Secure PostgreSQL replication
Support for TLS-secured PostgreSQL replication has been added. If you are
currently using PostgreSQL replication across the open internet without an
external means of securing the connection (e.g., a site-to-site VPN), then you
should immediately reconfigure your **primary** and **secondary** PostgreSQL instances
according to the [updated instructions][database].
If you *are* securing the connections externally and wish to continue doing so,
ensure you include the new option `--sslmode=prefer` in future invocations of
`gitlab-ctl replicate-geo-database`.
### HTTPS repository sync
Support for replicating repositories and wikis over HTTP/HTTPS has been added.
Replicating over SSH has been deprecated, and support for this option will be
removed in a future release.
To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit
**Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed,
press the "Edit" button, change the "Repository cloning" setting from
"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take
effect immediately.
Any new secondaries should be created using HTTP/HTTPS replication - this is the
default setting.
After you've verified that HTTP/HTTPS replication is working, you should remove
the now-unused SSH keys from your secondaries, as they may cause problems if the
**secondary** node if ever promoted to a **primary** node:
1. **(secondary)** Login to **all** your **secondary** nodes and run:
```ruby
sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub
```
### Hashed Storage
CAUTION: **Warning:**
Hashed storage is in **Alpha**. It is considered experimental and not
production-ready. See [Hashed Storage] for more detail.
If you previously enabled Hashed Storage and migrated all your existing
projects to Hashed Storage, disabling hashed storage will not migrate projects
to their previous project based storage path. As such, once enabled and
migrated we recommend leaving Hashed Storage enabled.
## Upgrading to GitLab 10.1
CAUTION: **Warning:**
Hashed storage is in **Alpha**. It is considered experimental and not
production-ready. See [Hashed Storage] for more detail.
[Hashed storage] was introduced in GitLab 10.0, and a [migration path][hashed-migration]
for existing repositories was added in GitLab 10.1.
## Upgrading to GitLab 10.0
Since GitLab 10.0, we require all **Geo** systems to [use SSH key lookups via
the database][ssh-fast-lookup] to avoid having to maintain consistency of the
`authorized_keys` file for SSH access. Failing to do this will prevent users
from being able to clone via SSH.
Note that in older versions of Geo, attachments downloaded on the **secondary**
nodes would be saved to the wrong directory. We recommend that you do the
following to clean this up.
On the **secondary** Geo nodes, run as root:
```sh
mv /var/opt/gitlab/gitlab-rails/working /var/opt/gitlab/gitlab-rails/working.old
mkdir /var/opt/gitlab/gitlab-rails/working
chmod 700 /var/opt/gitlab/gitlab-rails/working
chown git:git /var/opt/gitlab/gitlab-rails/working
```
You may delete `/var/opt/gitlab/gitlab-rails/working.old` any time.
Once this is done, we advise restarting GitLab on the **secondary** nodes for the
new working directory to be used:
```sh
sudo gitlab-ctl restart
```
## Upgrading from GitLab 9.3 or older
If you started running Geo on GitLab 9.3 or older, we recommend that you
resync your **secondary** PostgreSQL databases to use replication slots. If you
started using Geo with GitLab 9.4 or 10.x, no further action should be
required because replication slots are used by default. However, if you
started with GitLab 9.3 and upgraded later, you should still follow the
instructions below.
When in doubt, it does not hurt to do a resync. The easiest way to do this in
Omnibus is the following:
1. Make sure you have Omnibus GitLab on the **primary** server.
1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database.
1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`.
1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations.
1. Install GitLab on the **secondary** server.
1. Re-run the [database replication process](database.md#step-3-initiate-the-replication-process).
## Special update notes for 9.0.x
> **IMPORTANT**:
With GitLab 9.0, the PostgreSQL version is upgraded to 9.6 and manual steps are
required in order to update the **secondary** nodes and keep the Streaming
Replication working. Downtime is required, so plan ahead.
The following steps apply only if you upgrade from a 8.17 GitLab version to
9.0+. For previous versions, update to GitLab 8.17 first before attempting to
upgrade to 9.0+.
---
Make sure to follow the steps in the exact order as they appear below and pay
extra attention in what node (either **primary** or **secondary**) you execute them! Each step
is prepended with the relevant node for better clarity:
1. **(secondary)** Login to **all** your **secondary** nodes and stop all services:
```ruby
sudo gitlab-ctl stop
```
1. **(secondary)** Make a backup of the `recovery.conf` file on **all**
**secondary** nodes to preserve PostgreSQL's credentials:
```sh
sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/
```
1. **(primary)** Update the **primary** node to GitLab 9.0 following the
[regular update docs][update]. At the end of the update, the **primary** node
will be running with PostgreSQL 9.6.
1. **(primary)** To prevent a de-synchronization of the repository replication,
stop all services except `postgresql` as we will use it to re-initialize the
**secondary** node's database:
```sh
sudo gitlab-ctl stop
sudo gitlab-ctl start postgresql
```
1. **(secondary)** Run the following steps on each of the **secondary** nodes:
1. **(secondary)** Stop all services:
```sh
sudo gitlab-ctl stop
```
1. **(secondary)** Prevent running database migrations:
```sh
sudo touch /etc/gitlab/skip-auto-migrations
```
1. **(secondary)** Move the old database to another directory:
```sh
sudo mv /var/opt/gitlab/postgresql{,.bak}
```
1. **(secondary)** Update to GitLab 9.0 following the [regular update docs][update].
At the end of the update, the node will be running with PostgreSQL 9.6.
1. **(secondary)** Make sure all services are up:
```sh
sudo gitlab-ctl start
```
1. **(secondary)** Reconfigure GitLab:
```sh
sudo gitlab-ctl reconfigure
```
1. **(secondary)** Run the PostgreSQL upgrade command:
```sh
sudo gitlab-ctl pg-upgrade
```
1. **(secondary)** See the stored credentials for the database that you will
need to re-initialize the replication:
```sh
sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf
```
1. **(secondary)** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the
embedded paths if necessary:
```
#!/bin/bash
PORT="5432"
USER="gitlab_replicator"
echo ---------------------------------------------------------------
echo WARNING: Make sure this script is run from the secondary server
echo ---------------------------------------------------------------
echo
echo Enter the IP or FQDN of the primary PostgreSQL server
read HOST
echo Enter the password for $USER@$HOST
read -s PASSWORD
echo Enter the required sslmode
read SSLMODE
echo Stopping PostgreSQL and all GitLab services
sudo service gitlab stop
sudo service postgresql stop
echo Backing up postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
echo Cleaning up old cluster directory
sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data
echo Starting base backup as the replicator user
echo Enter the password for $USER@$HOST
sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
echo Writing recovery.conf file
sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE'
_EOF1_
"
echo Restoring postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
echo Starting PostgreSQL
sudo service postgresql start
```
1. **(secondary)** Run the recovery script using the credentials from the
previous step:
```sh
sudo bash /tmp/replica.sh
```
1. **(secondary)** Reconfigure GitLab:
```sh
sudo gitlab-ctl reconfigure
```
1. **(secondary)** Start all services:
```sh
sudo gitlab-ctl start
```
1. **(secondary)** Repeat the steps for the remaining **secondary** nodes.
1. **(primary)** After all **secondary** nodes are updated, start all services in
**primary** node:
```sh
sudo gitlab-ctl start
```
### Update tracking database on **secondary** node
After updating a **secondary** node, you might need to run migrations on
the tracking database. The tracking database was added in GitLab 9.1,
and it is required since 10.0.
1. Run database migrations on tracking database:
```sh
sudo gitlab-rake geo:db:migrate
```
1. Repeat this step for each **secondary** node.
[update]: ../../../update/README.md
[database]: database.md
[Hashed Storage]: ../../repository_storage_types.md
[hashed-migration]: ../../raketasks/storage.md
[ssh-fast-lookup]: ../../operations/fast_ssh_key_lookup.md

426
doc/administration/geo/replication/version_specific_updates.md Normal file
View file

@ -0,0 +1,426 @@
# Version specific update instructions
Check this document if it includes instructions for the version you are updating.
These steps go together with the [general steps](updating_the_geo_nodes.md#general-update-steps)
for updating Geo nodes.
## Updating to GitLab 12.1
By default, GitLab 12.1 will attempt to automatically update the
embedded PostgreSQL server to 10.7 from 9.6. Please see
[the omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance)
for the recommended procedure.
This can be temporarily disabled by running the following before updating:
```sh
sudo touch /etc/gitlab/disable-postgresql-upgrade
```
## Updating to GitLab 10.8
Before 10.8, broadcast messages would not propagate without flushing
the cache on the **secondary** nodes. This has been fixed in 10.8, but
requires one last cache flush on each **secondary** node:
```sh
sudo gitlab-rake cache:clear
```
## Updating to GitLab 10.6
In 10.4, we started to recommend that you define a password for database user (`gitlab`).
We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize
the Geo Tracking Database. We are also improving security by disabling the use of **trust**
authentication method.
1. **(primary)** Login to your **primary** node and run:
```sh
gitlab-ctl pg-password-md5 gitlab
# Enter password: <your_password_here>
# Confirm password: <your_password_here>
# fca0b89a972d69f00eb3ec98a5838484
```
Copy the generated hash and edit `/etc/gitlab/gitlab.rb`:
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below. If you have a high-availability setup, this
# must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'
```
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
```ruby
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this
```
1. **(primary)** Reconfigure and restart:
```sh
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
```
1. **(secondary)** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`:
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below. If you have a high-availability setup, this
# must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'
# Enable Foreign Data Wrapper
geo_secondary['db_fdw'] = true
# Secondary address in CIDR format, for example '5.6.7.8/32'
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
```
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
```ruby
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this
```
1. **(secondary)** Reconfigure and restart:
```sh
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
```
## Updating to GitLab 10.5
For Geo Disaster Recovery to work with minimum downtime, your **secondary** node
should use the same set of secrets as the **primary** node. However, setup instructions
prior to the 10.5 release only synchronized the `db_key_base` secret.
To rectify this error on existing installations, you should **overwrite** the
contents of `/etc/gitlab/gitlab-secrets.json` on each **secondary** node with the
contents of `/etc/gitlab/gitlab-secrets.json` on the **primary** node, then run the
following command on each **secondary** node:
```sh
sudo gitlab-ctl reconfigure
```
If you do not perform this step, you may find that two-factor authentication
[is broken following DR](../disaster_recovery/index.html#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken).
To prevent SSH requests to the newly promoted **primary** node from failing
due to SSH host key mismatch when updating the **primary** node domain's DNS record
you should perform the step to [Manually replicate **primary** SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys) in each
**secondary** node.
## Updating to GitLab 10.3
### Support for SSH repository synchronization removed
In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3,
support is removed entirely. All installations will switch to the HTTP/HTTPS
cloning method instead. Before updating, ensure that all your Geo nodes are
configured to use this method and that it works for your installation. In
particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-6-enable-git-access-over-httphttps).
Synchronizing repositories over the public Internet using HTTP is insecure, so
you should ensure that you have HTTPS configured before updating. Note that
file synchronization is **also** insecure in these cases!
## Updating to GitLab 10.2
### Secure PostgreSQL replication
Support for TLS-secured PostgreSQL replication has been added. If you are
currently using PostgreSQL replication across the open internet without an
external means of securing the connection (e.g., a site-to-site VPN), then you
should immediately reconfigure your **primary** and **secondary** PostgreSQL instances
according to the [updated instructions](database.md).
If you *are* securing the connections externally and wish to continue doing so,
ensure you include the new option `--sslmode=prefer` in future invocations of
`gitlab-ctl replicate-geo-database`.
### HTTPS repository sync
Support for replicating repositories and wikis over HTTP/HTTPS has been added.
Replicating over SSH has been deprecated, and support for this option will be
removed in a future release.
To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit
**Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed,
press the "Edit" button, change the "Repository cloning" setting from
"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take
effect immediately.
Any new secondaries should be created using HTTP/HTTPS replication - this is the
default setting.
After you've verified that HTTP/HTTPS replication is working, you should remove
the now-unused SSH keys from your secondaries, as they may cause problems if the
**secondary** node if ever promoted to a **primary** node:
1. **(secondary)** Login to **all** your **secondary** nodes and run:
```ruby
sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub
```
### Hashed Storage
CAUTION: **Warning:**
Hashed storage is in **Alpha**. It is considered experimental and not
production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail.
If you previously enabled Hashed Storage and migrated all your existing
projects to Hashed Storage, disabling hashed storage will not migrate projects
to their previous project based storage path. As such, once enabled and
migrated we recommend leaving Hashed Storage enabled.
## Updating to GitLab 10.1
CAUTION: **Warning:**
Hashed storage is in **Alpha**. It is considered experimental and not
production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail.
[Hashed storage](../../repository_storage_types.md) was introduced in
GitLab 10.0, and a [migration path](../../raketasks/storage.md) for
existing repositories was added in GitLab 10.1.
## Updating to GitLab 10.0
Since GitLab 10.0, we require all **Geo** systems to [use SSH key lookups via
the database](../../operations/fast_ssh_key_lookup.md) to avoid having to maintain consistency of the
`authorized_keys` file for SSH access. Failing to do this will prevent users
from being able to clone via SSH.
Note that in older versions of Geo, attachments downloaded on the **secondary**
nodes would be saved to the wrong directory. We recommend that you do the
following to clean this up.
On the **secondary** Geo nodes, run as root:
```sh
mv /var/opt/gitlab/gitlab-rails/working /var/opt/gitlab/gitlab-rails/working.old
mkdir /var/opt/gitlab/gitlab-rails/working
chmod 700 /var/opt/gitlab/gitlab-rails/working
chown git:git /var/opt/gitlab/gitlab-rails/working
```
You may delete `/var/opt/gitlab/gitlab-rails/working.old` any time.
Once this is done, we advise restarting GitLab on the **secondary** nodes for the
new working directory to be used:
```sh
sudo gitlab-ctl restart
```
## Updating from GitLab 9.3 or older
If you started running Geo on GitLab 9.3 or older, we recommend that you
resync your **secondary** PostgreSQL databases to use replication slots. If you
started using Geo with GitLab 9.4 or 10.x, no further action should be
required because replication slots are used by default. However, if you
started with GitLab 9.3 and updated later, you should still follow the
instructions below.
When in doubt, it does not hurt to do a resync. The easiest way to do this in
Omnibus is the following:
1. Make sure you have Omnibus GitLab on the **primary** server.
1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database.
1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`.
1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations.
1. Install GitLab on the **secondary** server.
1. Re-run the [database replication process](database.md#step-3-initiate-the-replication-process).
## Updating to GitLab 9.0
> **IMPORTANT**:
With GitLab 9.0, the PostgreSQL version is updated to 9.6 and manual steps are
required in order to update the **secondary** nodes and keep the Streaming
Replication working. Downtime is required, so plan ahead.
The following steps apply only if you update from a 8.17 GitLab version to
9.0+. For previous versions, update to GitLab 8.17 first before attempting to
update to 9.0+.
---
Make sure to follow the steps in the exact order as they appear below and pay
extra attention in what node (either **primary** or **secondary**) you execute them! Each step
is prepended with the relevant node for better clarity:
1. **(secondary)** Log in to **all** your **secondary** nodes and stop all services:
```ruby
sudo gitlab-ctl stop
```
1. **(secondary)** Make a backup of the `recovery.conf` file on **all**
**secondary** nodes to preserve PostgreSQL's credentials:
```sh
sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/
```
1. **(primary)** Update the **primary** node to GitLab 9.0 following the
[regular update docs](../../../update/README.md). At the end of the
update, the **primary** node will be running with PostgreSQL 9.6.
1. **(primary)** To prevent a de-synchronization of the repository replication,
stop all services except `postgresql` as we will use it to re-initialize the
**secondary** node's database:
```sh
sudo gitlab-ctl stop
sudo gitlab-ctl start postgresql
```
1. **(secondary)** Run the following steps on each of the **secondary** nodes:
1. **(secondary)** Stop all services:
```sh
sudo gitlab-ctl stop
```
1. **(secondary)** Prevent running database migrations:
```sh
sudo touch /etc/gitlab/skip-auto-migrations
```
1. **(secondary)** Move the old database to another directory:
```sh
sudo mv /var/opt/gitlab/postgresql{,.bak}
```
1. **(secondary)** Update to GitLab 9.0 following the [regular update docs](../../../update/README.md).
At the end of the update, the node will be running with PostgreSQL 9.6.
1. **(secondary)** Make sure all services are up:
```sh
sudo gitlab-ctl start
```
1. **(secondary)** Reconfigure GitLab:
```sh
sudo gitlab-ctl reconfigure
```
1. **(secondary)** Run the PostgreSQL upgrade command:
```sh
sudo gitlab-ctl pg-upgrade
```
1. **(secondary)** See the stored credentials for the database that you will
need to re-initialize the replication:
```sh
sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf
```
1. **(secondary)** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the
embedded paths if necessary:
```
#!/bin/bash
PORT="5432"
USER="gitlab_replicator"
echo ---------------------------------------------------------------
echo WARNING: Make sure this script is run from the secondary server
echo ---------------------------------------------------------------
echo
echo Enter the IP or FQDN of the primary PostgreSQL server
read HOST
echo Enter the password for $USER@$HOST
read -s PASSWORD
echo Enter the required sslmode
read SSLMODE
echo Stopping PostgreSQL and all GitLab services
sudo service gitlab stop
sudo service postgresql stop
echo Backing up postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
echo Cleaning up old cluster directory
sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data
echo Starting base backup as the replicator user
echo Enter the password for $USER@$HOST
sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
echo Writing recovery.conf file
sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE'
_EOF1_
"
echo Restoring postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
echo Starting PostgreSQL
sudo service postgresql start
```
1. **(secondary)** Run the recovery script using the credentials from the
previous step:
```sh
sudo bash /tmp/replica.sh
```
1. **(secondary)** Reconfigure GitLab:
```sh
sudo gitlab-ctl reconfigure
```
1. **(secondary)** Start all services:
```sh
sudo gitlab-ctl start
```
1. **(secondary)** Repeat the steps for the remaining **secondary** nodes.
1. **(primary)** After all **secondary** nodes are updated, start all services in
**primary** node:
```sh
sudo gitlab-ctl start
```
### Update tracking database on **secondary** node
After updating a **secondary** node, you might need to run migrations on
the tracking database. The tracking database was added in GitLab 9.1,
and it is required since 10.0.
1. Run database migrations on tracking database:
```sh
sudo gitlab-rake geo:db:migrate
```
1. Repeat this step for each **secondary** node.