Merge branch 'docs-geo-markdown-1' into 'master'
Docs: Clean up markdown spacing in geo docs See merge request gitlab-org/gitlab-ce!30112
This commit is contained in:
commit
f55d85707f
|
@ -143,11 +143,13 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch
|
|||
1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**:
|
||||
![Project admin page](img/checksum-differences-admin-project-page.png)
|
||||
|
||||
1. Navigate to the project's repository directory on both **primary** and **secondary** nodes (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure.
|
||||
1. Navigate to the project's repository directory on both **primary** and **secondary** nodes
|
||||
(the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs`
|
||||
is customized, check the directory layout on your server to be sure.
|
||||
|
||||
```sh
|
||||
cd /var/opt/gitlab/git-data/repositories
|
||||
```
|
||||
```sh
|
||||
cd /var/opt/gitlab/git-data/repositories
|
||||
```
|
||||
|
||||
1. Run the following command on the **primary** node, redirecting the output to a file:
|
||||
|
||||
|
|
|
@ -21,20 +21,20 @@ To bring the former **primary** node up to date:
|
|||
1. SSH into the former **primary** node that has fallen behind.
|
||||
1. Make sure all the services are up:
|
||||
|
||||
```sh
|
||||
sudo gitlab-ctl start
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-ctl start
|
||||
```
|
||||
|
||||
> **Note 1:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary],
|
||||
> you need to undo those steps now. For Debian/Ubuntu you just need to run
|
||||
> `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install
|
||||
> the GitLab instance from scratch and set it up as a **secondary** node by
|
||||
> following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step.
|
||||
>
|
||||
> **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record)
|
||||
> for this node during disaster recovery procedure you may need to [block
|
||||
> all the writes to this node](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/gitlab-geo/planned-failover.md#block-primary-traffic)
|
||||
> during this procedure.
|
||||
NOTE: **Note:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary],
|
||||
you need to undo those steps now. For Debian/Ubuntu you just need to run
|
||||
`sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install
|
||||
the GitLab instance from scratch and set it up as a **secondary** node by
|
||||
following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step.
|
||||
|
||||
NOTE: **Note:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record)
|
||||
for this node during disaster recovery procedure you may need to [block
|
||||
all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node)
|
||||
during this procedure.
|
||||
|
||||
1. [Setup database replication][database-replication]. Note that in this
|
||||
case, **primary** node refers to the current **primary** node, and **secondary** node refers to the
|
||||
|
|
|
@ -39,48 +39,50 @@ must disable the **primary** node.
|
|||
|
||||
1. SSH into the **primary** node to stop and disable GitLab, if possible:
|
||||
|
||||
```sh
|
||||
sudo gitlab-ctl stop
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-ctl stop
|
||||
```
|
||||
|
||||
Prevent GitLab from starting up again if the server unexpectedly reboots:
|
||||
Prevent GitLab from starting up again if the server unexpectedly reboots:
|
||||
|
||||
```sh
|
||||
sudo systemctl disable gitlab-runsvdir
|
||||
```
|
||||
```sh
|
||||
sudo systemctl disable gitlab-runsvdir
|
||||
```
|
||||
|
||||
> **CentOS only**: In CentOS 6 or older, there is no easy way to prevent GitLab from being
|
||||
> started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]).
|
||||
> It may be safest to uninstall the GitLab package completely:
|
||||
NOTE: **Note:**
|
||||
(**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being
|
||||
started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]).
|
||||
It may be safest to uninstall the GitLab package completely:
|
||||
|
||||
```sh
|
||||
yum remove gitlab-ee
|
||||
```
|
||||
```sh
|
||||
yum remove gitlab-ee
|
||||
```
|
||||
|
||||
> **Ubuntu 14.04 LTS**: If you are using an older version of Ubuntu
|
||||
> or any other distro based on the Upstart init system, you can prevent GitLab
|
||||
> from starting if the machine reboots by doing the following:
|
||||
NOTE: **Note:**
|
||||
(**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu
|
||||
or any other distro based on the Upstart init system, you can prevent GitLab
|
||||
from starting if the machine reboots by doing the following:
|
||||
|
||||
```sh
|
||||
initctl stop gitlab-runsvvdir
|
||||
echo 'manual' > /etc/init/gitlab-runsvdir.override
|
||||
initctl reload-configuration
|
||||
```
|
||||
```sh
|
||||
initctl stop gitlab-runsvvdir
|
||||
echo 'manual' > /etc/init/gitlab-runsvdir.override
|
||||
initctl reload-configuration
|
||||
```
|
||||
|
||||
1. If you do not have SSH access to the **primary** node, take the machine offline and
|
||||
prevent it from rebooting by any means at your disposal.
|
||||
Since there are many ways you may prefer to accomplish this, we will avoid a
|
||||
single recommendation. You may need to:
|
||||
- Reconfigure the load balancers.
|
||||
- Change DNS records (e.g., point the primary DNS record to the **secondary**
|
||||
node in order to stop usage of the **primary** node).
|
||||
- Stop the virtual servers.
|
||||
- Block traffic through a firewall.
|
||||
- Revoke object storage permissions from the **primary** node.
|
||||
- Physically disconnect a machine.
|
||||
prevent it from rebooting by any means at your disposal.
|
||||
Since there are many ways you may prefer to accomplish this, we will avoid a
|
||||
single recommendation. You may need to:
|
||||
|
||||
1. If you plan to
|
||||
[update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record),
|
||||
- Reconfigure the load balancers.
|
||||
- Change DNS records (e.g., point the primary DNS record to the **secondary**
|
||||
node in order to stop usage of the **primary** node).
|
||||
- Stop the virtual servers.
|
||||
- Block traffic through a firewall.
|
||||
- Revoke object storage permissions from the **primary** node.
|
||||
- Physically disconnect a machine.
|
||||
|
||||
1. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record),
|
||||
you may wish to lower the TTL now to speed up propagation.
|
||||
|
||||
### Step 3. Promoting a **secondary** node
|
||||
|
@ -94,26 +96,26 @@ the **secondary** to the **primary**.
|
|||
|
||||
1. SSH in to your **secondary** node and login as root:
|
||||
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
|
||||
1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by
|
||||
removing any lines that enabled the `geo_secondary_role`:
|
||||
|
||||
```ruby
|
||||
## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
|
||||
geo_secondary_role['enable'] = true
|
||||
```ruby
|
||||
## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
|
||||
geo_secondary_role['enable'] = true
|
||||
|
||||
## In 11.5+ documentation, the role was enabled as follows. Remove this line.
|
||||
roles ['geo_secondary_role']
|
||||
```
|
||||
## In 11.5+ documentation, the role was enabled as follows. Remove this line.
|
||||
roles ['geo_secondary_role']
|
||||
```
|
||||
|
||||
1. Promote the **secondary** node to the **primary** node. Execute:
|
||||
|
||||
```sh
|
||||
gitlab-ctl promote-to-primary-node
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl promote-to-primary-node
|
||||
```
|
||||
|
||||
1. Verify you can connect to the newly promoted **primary** node using the URL used
|
||||
previously for the **secondary** node.
|
||||
|
@ -129,31 +131,31 @@ do this manually.
|
|||
1. SSH in to the database node in the **secondary** and trigger PostgreSQL to
|
||||
promote to read-write:
|
||||
|
||||
```bash
|
||||
sudo gitlab-pg-ctl promote
|
||||
```
|
||||
```bash
|
||||
sudo gitlab-pg-ctl promote
|
||||
```
|
||||
|
||||
1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to
|
||||
reflect its new status as **primary** by removing any lines that enabled the
|
||||
`geo_secondary_role`:
|
||||
|
||||
```ruby
|
||||
## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
|
||||
geo_secondary_role['enable'] = true
|
||||
```ruby
|
||||
## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
|
||||
geo_secondary_role['enable'] = true
|
||||
|
||||
## In 11.5+ documentation, the role was enabled as follows. Remove this line.
|
||||
roles ['geo_secondary_role']
|
||||
```
|
||||
## In 11.5+ documentation, the role was enabled as follows. Remove this line.
|
||||
roles ['geo_secondary_role']
|
||||
```
|
||||
|
||||
After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each
|
||||
machine so the changes take effect.
|
||||
After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each
|
||||
machine so the changes take effect.
|
||||
|
||||
1. Promote the **secondary** to **primary**. SSH into a single application
|
||||
server and execute:
|
||||
|
||||
```bash
|
||||
sudo gitlab-rake geo:set_secondary_as_primary
|
||||
```
|
||||
```bash
|
||||
sudo gitlab-rake geo:set_secondary_as_primary
|
||||
```
|
||||
|
||||
1. Verify you can connect to the newly promoted **primary** using the URL used
|
||||
previously for the **secondary**.
|
||||
|
@ -167,37 +169,37 @@ secondary domain, like changing Git remotes and API URLs.
|
|||
|
||||
1. SSH into the **secondary** node and login as root:
|
||||
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
|
||||
1. Update the primary domain's DNS record. After updating the primary domain's
|
||||
DNS records to point to the **secondary** node, edit `/etc/gitlab/gitlab.rb` on the
|
||||
**secondary** node to reflect the new URL:
|
||||
|
||||
```ruby
|
||||
# Change the existing external_url configuration
|
||||
external_url 'https://<new_external_url>'
|
||||
```
|
||||
```ruby
|
||||
# Change the existing external_url configuration
|
||||
external_url 'https://<new_external_url>'
|
||||
```
|
||||
|
||||
NOTE: **Note**
|
||||
Changing `external_url` won't prevent access via the old secondary URL, as
|
||||
long as the secondary DNS records are still intact.
|
||||
NOTE: **Note**
|
||||
Changing `external_url` won't prevent access via the old secondary URL, as
|
||||
long as the secondary DNS records are still intact.
|
||||
|
||||
1. Reconfigure the **secondary** node for the change to take effect:
|
||||
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
|
||||
1. Execute the command below to update the newly promoted **primary** node URL:
|
||||
|
||||
```sh
|
||||
gitlab-rake geo:update_primary_node_url
|
||||
```
|
||||
```sh
|
||||
gitlab-rake geo:update_primary_node_url
|
||||
```
|
||||
|
||||
This command will use the changed `external_url` configuration defined
|
||||
in `/etc/gitlab/gitlab.rb`.
|
||||
This command will use the changed `external_url` configuration defined
|
||||
in `/etc/gitlab/gitlab.rb`.
|
||||
|
||||
1. Verify you can connect to the newly promoted **primary** using its URL.
|
||||
If you updated the DNS records for the primary domain, these changes may
|
||||
|
@ -231,62 +233,61 @@ and after that you also need two extra steps.
|
|||
|
||||
1. SSH into the new **primary** node and login as root:
|
||||
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
|
||||
1. Edit `/etc/gitlab/gitlab.rb`
|
||||
|
||||
```ruby
|
||||
## Enable a Geo Primary role (if you haven't yet)
|
||||
roles ['geo_primary_role']
|
||||
```ruby
|
||||
## Enable a Geo Primary role (if you haven't yet)
|
||||
roles ['geo_primary_role']
|
||||
|
||||
##
|
||||
# Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be
|
||||
# public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32']
|
||||
##
|
||||
postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32']
|
||||
##
|
||||
# Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be
|
||||
# public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32']
|
||||
##
|
||||
postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32']
|
||||
|
||||
# Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have
|
||||
postgresql['max_replication_slots'] = 1
|
||||
# Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have
|
||||
postgresql['max_replication_slots'] = 1
|
||||
|
||||
##
|
||||
## Disable automatic database migrations temporarily
|
||||
## (until PostgreSQL is restarted and listening on the private address).
|
||||
##
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
##
|
||||
## Disable automatic database migrations temporarily
|
||||
## (until PostgreSQL is restarted and listening on the private address).
|
||||
##
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
```
|
||||
|
||||
```
|
||||
|
||||
(For more details about these settings you can read [Configure the primary server][configure-the-primary-server])
|
||||
(For more details about these settings you can read [Configure the primary server][configure-the-primary-server])
|
||||
|
||||
1. Save the file and reconfigure GitLab for the database listen changes and
|
||||
the replication slot changes to be applied.
|
||||
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
|
||||
Restart PostgreSQL for its changes to take effect:
|
||||
Restart PostgreSQL for its changes to take effect:
|
||||
|
||||
```sh
|
||||
gitlab-ctl restart postgresql
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl restart postgresql
|
||||
```
|
||||
|
||||
1. Re-enable migrations now that PostgreSQL is restarted and listening on the
|
||||
private address.
|
||||
|
||||
Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`:
|
||||
Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`:
|
||||
|
||||
```ruby
|
||||
gitlab_rails['auto_migrate'] = true
|
||||
```
|
||||
```ruby
|
||||
gitlab_rails['auto_migrate'] = true
|
||||
```
|
||||
|
||||
Save the file and reconfigure GitLab:
|
||||
Save the file and reconfigure GitLab:
|
||||
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
|
||||
### Step 2. Initiate the replication process
|
||||
|
||||
|
|
|
@ -143,26 +143,26 @@ access to the **primary** node during the maintenance window.
|
|||
all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and
|
||||
the **secondary** node's IP.
|
||||
|
||||
For instance, you might run the following commands on the server(s) making up your **primary** node:
|
||||
For instance, you might run the following commands on the server(s) making up your **primary** node:
|
||||
|
||||
```sh
|
||||
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT
|
||||
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT
|
||||
sudo iptables -A INPUT --destination-port 22 -j REJECT
|
||||
```sh
|
||||
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT
|
||||
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT
|
||||
sudo iptables -A INPUT --destination-port 22 -j REJECT
|
||||
|
||||
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT
|
||||
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT
|
||||
sudo iptables -A INPUT --tcp-dport 80 -j REJECT
|
||||
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT
|
||||
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT
|
||||
sudo iptables -A INPUT --tcp-dport 80 -j REJECT
|
||||
|
||||
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT
|
||||
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT
|
||||
sudo iptables -A INPUT --tcp-dport 443 -j REJECT
|
||||
```
|
||||
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT
|
||||
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT
|
||||
sudo iptables -A INPUT --tcp-dport 443 -j REJECT
|
||||
```
|
||||
|
||||
From this point, users will be unable to view their data or make changes on the
|
||||
**primary** node. They will also be unable to log in to the **secondary** node.
|
||||
However, existing sessions will work for the remainder of the maintenance period, and
|
||||
public data will be accessible throughout.
|
||||
From this point, users will be unable to view their data or make changes on the
|
||||
**primary** node. They will also be unable to log in to the **secondary** node.
|
||||
However, existing sessions will work for the remainder of the maintenance period, and
|
||||
public data will be accessible throughout.
|
||||
|
||||
1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via
|
||||
another IP. The server should refuse connection.
|
||||
|
@ -187,10 +187,11 @@ access to the **primary** node during the maintenance window.
|
|||
before it is completed will cause the work to be lost.
|
||||
1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the
|
||||
following conditions to be true of the **secondary** node you are failing over to:
|
||||
- All replication meters to each 100% replicated, 0% failures.
|
||||
- All verification meters reach 100% verified, 0% failures.
|
||||
- Database replication lag is 0ms.
|
||||
- The Geo log cursor is up to date (0 events behind).
|
||||
|
||||
- All replication meters to each 100% replicated, 0% failures.
|
||||
- All verification meters reach 100% verified, 0% failures.
|
||||
- Database replication lag is 0ms.
|
||||
- The Geo log cursor is up to date (0 events behind).
|
||||
|
||||
1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues**
|
||||
and wait for all the `geo` queues to drop to 0 queued and 0 running jobs.
|
||||
|
|
|
@ -16,11 +16,10 @@ The basic steps of configuring a **secondary** node are to:
|
|||
You are encouraged to first read through all the steps before executing them
|
||||
in your testing/production environment.
|
||||
|
||||
> **Notes:**
|
||||
> - **Do not** setup any custom authentication for the **secondary** nodes. This will be
|
||||
handled by the **primary** node.
|
||||
> - Any change that requires access to the **Admin Area** needs to be done in the
|
||||
**primary** node because the **secondary** node is a read-only replica.
|
||||
NOTE: **Note:**
|
||||
**Do not** set up any custom authentication for the **secondary** nodes. This will be handled by the **primary** node.
|
||||
Any change that requires access to the **Admin Area** needs to be done in the
|
||||
**primary** node because the **secondary** node is a read-only replica.
|
||||
|
||||
### Step 1. Manually replicate secret GitLab values
|
||||
|
||||
|
@ -31,47 +30,47 @@ they must be manually replicated to the **secondary** node.
|
|||
|
||||
1. SSH into the **primary** node, and execute the command below:
|
||||
|
||||
```sh
|
||||
sudo cat /etc/gitlab/gitlab-secrets.json
|
||||
```
|
||||
```sh
|
||||
sudo cat /etc/gitlab/gitlab-secrets.json
|
||||
```
|
||||
|
||||
This will display the secrets that need to be replicated, in JSON format.
|
||||
This will display the secrets that need to be replicated, in JSON format.
|
||||
|
||||
1. SSH into the **secondary** node and login as the `root` user:
|
||||
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
|
||||
1. Make a backup of any existing secrets:
|
||||
|
||||
```sh
|
||||
mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F`
|
||||
```
|
||||
```sh
|
||||
mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F`
|
||||
```
|
||||
|
||||
1. Copy `/etc/gitlab/gitlab-secrets.json` from the **primary** node to the **secondary** node, or
|
||||
copy-and-paste the file contents between nodes:
|
||||
|
||||
```sh
|
||||
sudo editor /etc/gitlab/gitlab-secrets.json
|
||||
```sh
|
||||
sudo editor /etc/gitlab/gitlab-secrets.json
|
||||
|
||||
# paste the output of the `cat` command you ran on the primary
|
||||
# save and exit
|
||||
```
|
||||
# paste the output of the `cat` command you ran on the primary
|
||||
# save and exit
|
||||
```
|
||||
|
||||
1. Ensure the file permissions are correct:
|
||||
|
||||
```sh
|
||||
chown root:root /etc/gitlab/gitlab-secrets.json
|
||||
chmod 0600 /etc/gitlab/gitlab-secrets.json
|
||||
```
|
||||
```sh
|
||||
chown root:root /etc/gitlab/gitlab-secrets.json
|
||||
chmod 0600 /etc/gitlab/gitlab-secrets.json
|
||||
```
|
||||
|
||||
1. Reconfigure the **secondary** node for the change to take effect:
|
||||
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
gitlab-ctl restart
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
gitlab-ctl restart
|
||||
```
|
||||
|
||||
### Step 2. Manually replicate the **primary** node's SSH host keys
|
||||
|
||||
|
@ -89,80 +88,80 @@ keys must be manually replicated to the **secondary** node.
|
|||
|
||||
1. SSH into the **secondary** node and login as the `root` user:
|
||||
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
|
||||
1. Make a backup of any existing SSH host keys:
|
||||
|
||||
```sh
|
||||
find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \;
|
||||
```
|
||||
```sh
|
||||
find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \;
|
||||
```
|
||||
|
||||
1. Copy OpenSSH host keys from the **primary** node:
|
||||
|
||||
If you can access your **primary** node using the **root** user:
|
||||
If you can access your **primary** node using the **root** user:
|
||||
|
||||
```sh
|
||||
# Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server
|
||||
scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh
|
||||
```
|
||||
```sh
|
||||
# Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server
|
||||
scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh
|
||||
```
|
||||
|
||||
If you only have access through a user with **sudo** privileges:
|
||||
If you only have access through a user with **sudo** privileges:
|
||||
|
||||
```sh
|
||||
# Run this from your primary node:
|
||||
sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key*
|
||||
```sh
|
||||
# Run this from your primary node:
|
||||
sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key*
|
||||
|
||||
# Run this from your secondary node:
|
||||
scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz .
|
||||
tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh
|
||||
```
|
||||
# Run this from your secondary node:
|
||||
scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz .
|
||||
tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh
|
||||
```
|
||||
|
||||
1. On your **secondary** node, ensure the file permissions are correct:
|
||||
|
||||
```sh
|
||||
chown root:root /etc/ssh/ssh_host_*_key*
|
||||
chmod 0600 /etc/ssh/ssh_host_*_key*
|
||||
```
|
||||
```sh
|
||||
chown root:root /etc/ssh/ssh_host_*_key*
|
||||
chmod 0600 /etc/ssh/ssh_host_*_key*
|
||||
```
|
||||
|
||||
1. To verify key fingerprint matches, execute the following command on both nodes:
|
||||
|
||||
```sh
|
||||
for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done
|
||||
```
|
||||
```sh
|
||||
for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done
|
||||
```
|
||||
|
||||
You should get an output similar to this one and they should be identical on both nodes:
|
||||
You should get an output similar to this one and they should be identical on both nodes:
|
||||
|
||||
```sh
|
||||
1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA)
|
||||
256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA)
|
||||
256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519)
|
||||
2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA)
|
||||
```
|
||||
```sh
|
||||
1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA)
|
||||
256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA)
|
||||
256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519)
|
||||
2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA)
|
||||
```
|
||||
|
||||
1. Verify that you have the correct public keys for the existing private keys:
|
||||
|
||||
```sh
|
||||
# This will print the fingerprint for private keys:
|
||||
for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done
|
||||
```sh
|
||||
# This will print the fingerprint for private keys:
|
||||
for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done
|
||||
|
||||
# This will print the fingerprint for public keys:
|
||||
for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done
|
||||
```
|
||||
# This will print the fingerprint for public keys:
|
||||
for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done
|
||||
```
|
||||
|
||||
NOTE: **Note**:
|
||||
The output for private keys and public keys command should generate the same fingerprint.
|
||||
NOTE: **Note:**
|
||||
The output for private keys and public keys command should generate the same fingerprint.
|
||||
|
||||
1. Restart sshd on your **secondary** node:
|
||||
|
||||
```sh
|
||||
# Debian or Ubuntu installations
|
||||
sudo service ssh reload
|
||||
```sh
|
||||
# Debian or Ubuntu installations
|
||||
sudo service ssh reload
|
||||
|
||||
# CentOS installations
|
||||
sudo service sshd reload
|
||||
```
|
||||
# CentOS installations
|
||||
sudo service sshd reload
|
||||
```
|
||||
|
||||
### Step 3. Add the **secondary** node
|
||||
|
||||
|
@ -176,22 +175,22 @@ keys must be manually replicated to the **secondary** node.
|
|||
1. Click the **Add node** button.
|
||||
1. SSH into your GitLab **secondary** server and restart the services:
|
||||
|
||||
```sh
|
||||
gitlab-ctl restart
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl restart
|
||||
```
|
||||
|
||||
Check if there are any common issue with your Geo setup by running:
|
||||
Check if there are any common issue with your Geo setup by running:
|
||||
|
||||
```sh
|
||||
gitlab-rake gitlab:geo:check
|
||||
```
|
||||
```sh
|
||||
gitlab-rake gitlab:geo:check
|
||||
```
|
||||
|
||||
1. SSH into your **primary** server and login as root to verify the
|
||||
**secondary** node is reachable or there are any common issue with your Geo setup:
|
||||
|
||||
```sh
|
||||
gitlab-rake gitlab:geo:check
|
||||
```
|
||||
```sh
|
||||
gitlab-rake gitlab:geo:check
|
||||
```
|
||||
|
||||
Once added to the admin panel and restarted, the **secondary** node will automatically start
|
||||
replicating missing data from the **primary** node in a process known as **backfill**.
|
||||
|
@ -250,9 +249,8 @@ The two most obvious issues that can become apparent in the dashboard are:
|
|||
1. Database replication not working well.
|
||||
1. Instance to instance notification not working. In that case, it can be
|
||||
something of the following:
|
||||
- You are using a custom certificate or custom CA (see the
|
||||
[troubleshooting document]).
|
||||
- The instance is firewalled (check your firewall rules).
|
||||
- You are using a custom certificate or custom CA (see the [troubleshooting document](troubleshooting.md)).
|
||||
- The instance is firewalled (check your firewall rules).
|
||||
|
||||
Please note that disabling a **secondary** node will stop the synchronization process.
|
||||
|
||||
|
@ -304,5 +302,4 @@ See the [troubleshooting document](troubleshooting.md).
|
|||
[gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789
|
||||
[gitlab-com/infrastructure#2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821
|
||||
[omnibus-ssl]: https://docs.gitlab.com/omnibus/settings/ssl.html
|
||||
[troubleshooting document]: troubleshooting.md
|
||||
[using-geo]: using_a_geo_server.md
|
||||
|
|
|
@ -52,186 +52,188 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
|
|||
|
||||
1. SSH into your GitLab **primary** server and login as root:
|
||||
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
|
||||
1. Execute the command below to define the node as **primary** node:
|
||||
|
||||
```sh
|
||||
gitlab-ctl set-geo-primary-node
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl set-geo-primary-node
|
||||
```
|
||||
|
||||
This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`.
|
||||
This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`.
|
||||
|
||||
1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined:
|
||||
|
||||
Generate a MD5 hash of the desired password:
|
||||
Generate a MD5 hash of the desired password:
|
||||
|
||||
```sh
|
||||
gitlab-ctl pg-password-md5 gitlab
|
||||
# Enter password: <your_password_here>
|
||||
# Confirm password: <your_password_here>
|
||||
# fca0b89a972d69f00eb3ec98a5838484
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl pg-password-md5 gitlab
|
||||
# Enter password: <your_password_here>
|
||||
# Confirm password: <your_password_here>
|
||||
# fca0b89a972d69f00eb3ec98a5838484
|
||||
```
|
||||
|
||||
Edit `/etc/gitlab/gitlab.rb`:
|
||||
Edit `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```ruby
|
||||
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
|
||||
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
|
||||
```ruby
|
||||
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
|
||||
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
|
||||
|
||||
# Every node that runs Unicorn or Sidekiq needs to have the database
|
||||
# password specified as below. If you have a high-availability setup, this
|
||||
# must be present in all application nodes.
|
||||
gitlab_rails['db_password'] = '<your_password_here>'
|
||||
```
|
||||
# Every node that runs Unicorn or Sidekiq needs to have the database
|
||||
# password specified as below. If you have a high-availability setup, this
|
||||
# must be present in all application nodes.
|
||||
gitlab_rails['db_password'] = '<your_password_here>'
|
||||
```
|
||||
|
||||
1. Omnibus GitLab already has a [replication user]
|
||||
called `gitlab_replicator`. You must set the password for this user manually.
|
||||
You will be prompted to enter a password:
|
||||
|
||||
```sh
|
||||
gitlab-ctl set-replication-password
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl set-replication-password
|
||||
```
|
||||
|
||||
This command will also read the `postgresql['sql_replication_user']` Omnibus
|
||||
setting in case you have changed `gitlab_replicator` username to something
|
||||
else.
|
||||
This command will also read the `postgresql['sql_replication_user']` Omnibus
|
||||
setting in case you have changed `gitlab_replicator` username to something
|
||||
else.
|
||||
|
||||
If you are using an external database not managed by Omnibus GitLab, you need
|
||||
to create the replicator user and define a password to it manually:
|
||||
If you are using an external database not managed by Omnibus GitLab, you need
|
||||
to create the replicator user and define a password to it manually:
|
||||
|
||||
```sql
|
||||
--- Create a new user 'replicator'
|
||||
CREATE USER gitlab_replicator;
|
||||
```sql
|
||||
--- Create a new user 'replicator'
|
||||
CREATE USER gitlab_replicator;
|
||||
|
||||
--- Set/change a password and grants replication privilege
|
||||
ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>';
|
||||
```
|
||||
--- Set/change a password and grants replication privilege
|
||||
ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>';
|
||||
```
|
||||
|
||||
1. Configure PostgreSQL to listen on network interfaces:
|
||||
|
||||
For security reasons, PostgreSQL does not listen on any network interfaces
|
||||
by default. However, Geo requires the **secondary** node to be able to
|
||||
connect to the **primary** node's database. For this reason, we need the address of
|
||||
each node. Note: For external PostgreSQL instances, see [additional instructions](external_database.md).
|
||||
For security reasons, PostgreSQL does not listen on any network interfaces
|
||||
by default. However, Geo requires the **secondary** node to be able to
|
||||
connect to the **primary** node's database. For this reason, we need the address of
|
||||
each node.
|
||||
|
||||
NOTE: **Note:** For external PostgreSQL instances, see [additional instructions](external_database.md).
|
||||
|
||||
If you are using a cloud provider, you can lookup the addresses for each
|
||||
Geo node through your cloud provider's management console.
|
||||
If you are using a cloud provider, you can lookup the addresses for each
|
||||
Geo node through your cloud provider's management console.
|
||||
|
||||
To lookup the address of a Geo node, SSH in to the Geo node and execute:
|
||||
To lookup the address of a Geo node, SSH in to the Geo node and execute:
|
||||
|
||||
```sh
|
||||
##
|
||||
## Private address
|
||||
##
|
||||
ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}'
|
||||
```sh
|
||||
##
|
||||
## Private address
|
||||
##
|
||||
ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}'
|
||||
|
||||
##
|
||||
## Public address
|
||||
##
|
||||
echo "External address: $(curl --silent ipinfo.io/ip)"
|
||||
```
|
||||
##
|
||||
## Public address
|
||||
##
|
||||
echo "External address: $(curl --silent ipinfo.io/ip)"
|
||||
```
|
||||
|
||||
In most cases, the following addresses will be used to configure GitLab
|
||||
Geo:
|
||||
In most cases, the following addresses will be used to configure GitLab
|
||||
Geo:
|
||||
|
||||
| Configuration | Address |
|
||||
|:----------------------------------------|:------------------------------------------------------|
|
||||
| `postgresql['listen_address']` | **Primary** node's public or VPC private address. |
|
||||
| `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. |
|
||||
| Configuration | Address |
|
||||
|:----------------------------------------|:------------------------------------------------------|
|
||||
| `postgresql['listen_address']` | **Primary** node's public or VPC private address. |
|
||||
| `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. |
|
||||
|
||||
If you are using Google Cloud Platform, SoftLayer, or any other vendor that
|
||||
provides a virtual private cloud (VPC) you can use the **secondary** node's private
|
||||
address (corresponds to "internal address" for Google Cloud Platform) for
|
||||
`postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`.
|
||||
If you are using Google Cloud Platform, SoftLayer, or any other vendor that
|
||||
provides a virtual private cloud (VPC) you can use the **secondary** node's private
|
||||
address (corresponds to "internal address" for Google Cloud Platform) for
|
||||
`postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`.
|
||||
|
||||
The `listen_address` option opens PostgreSQL up to network connections
|
||||
with the interface corresponding to the given address. See [the PostgreSQL
|
||||
documentation][pg-docs-runtime-conn] for more details.
|
||||
The `listen_address` option opens PostgreSQL up to network connections
|
||||
with the interface corresponding to the given address. See [the PostgreSQL
|
||||
documentation][pg-docs-runtime-conn] for more details.
|
||||
|
||||
Depending on your network configuration, the suggested addresses may not
|
||||
be correct. If your **primary** node and **secondary** nodes connect over a local
|
||||
area network, or a virtual network connecting availability zones like
|
||||
[Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/)
|
||||
you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`.
|
||||
Depending on your network configuration, the suggested addresses may not
|
||||
be correct. If your **primary** node and **secondary** nodes connect over a local
|
||||
area network, or a virtual network connecting availability zones like
|
||||
[Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/)
|
||||
you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`.
|
||||
|
||||
Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
|
||||
addresses with addresses appropriate to your network configuration:
|
||||
Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
|
||||
addresses with addresses appropriate to your network configuration:
|
||||
|
||||
```ruby
|
||||
##
|
||||
## Geo Primary role
|
||||
## - configure dependent flags automatically to enable Geo
|
||||
##
|
||||
roles ['geo_primary_role']
|
||||
```ruby
|
||||
##
|
||||
## Geo Primary role
|
||||
## - configure dependent flags automatically to enable Geo
|
||||
##
|
||||
roles ['geo_primary_role']
|
||||
|
||||
##
|
||||
## Primary address
|
||||
## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node
|
||||
##
|
||||
postgresql['listen_address'] = '<primary_node_ip>'
|
||||
##
|
||||
## Primary address
|
||||
## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node
|
||||
##
|
||||
postgresql['listen_address'] = '<primary_node_ip>'
|
||||
|
||||
##
|
||||
# Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be
|
||||
# public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32']
|
||||
##
|
||||
postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32']
|
||||
##
|
||||
# Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be
|
||||
# public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32']
|
||||
##
|
||||
postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32']
|
||||
|
||||
##
|
||||
## Replication settings
|
||||
## - set this to be the number of Geo secondary nodes you have
|
||||
##
|
||||
postgresql['max_replication_slots'] = 1
|
||||
# postgresql['max_wal_senders'] = 10
|
||||
# postgresql['wal_keep_segments'] = 10
|
||||
##
|
||||
## Replication settings
|
||||
## - set this to be the number of Geo secondary nodes you have
|
||||
##
|
||||
postgresql['max_replication_slots'] = 1
|
||||
# postgresql['max_wal_senders'] = 10
|
||||
# postgresql['wal_keep_segments'] = 10
|
||||
|
||||
##
|
||||
## Disable automatic database migrations temporarily
|
||||
## (until PostgreSQL is restarted and listening on the private address).
|
||||
##
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
```
|
||||
##
|
||||
## Disable automatic database migrations temporarily
|
||||
## (until PostgreSQL is restarted and listening on the private address).
|
||||
##
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
```
|
||||
|
||||
1. Optional: If you want to add another **secondary** node, the relevant setting would look like:
|
||||
|
||||
```ruby
|
||||
postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32']
|
||||
```
|
||||
```ruby
|
||||
postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32']
|
||||
```
|
||||
|
||||
You may also want to edit the `wal_keep_segments` and `max_wal_senders` to
|
||||
match your database replication requirements. Consult the [PostgreSQL -
|
||||
Replication documentation][pg-docs-runtime-replication]
|
||||
for more information.
|
||||
You may also want to edit the `wal_keep_segments` and `max_wal_senders` to
|
||||
match your database replication requirements. Consult the [PostgreSQL -
|
||||
Replication documentation][pg-docs-runtime-replication]
|
||||
for more information.
|
||||
|
||||
1. Save the file and reconfigure GitLab for the database listen changes and
|
||||
the replication slot changes to be applied:
|
||||
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
|
||||
Restart PostgreSQL for its changes to take effect:
|
||||
Restart PostgreSQL for its changes to take effect:
|
||||
|
||||
```sh
|
||||
gitlab-ctl restart postgresql
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl restart postgresql
|
||||
```
|
||||
|
||||
1. Re-enable migrations now that PostgreSQL is restarted and listening on the
|
||||
private address.
|
||||
|
||||
Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`:
|
||||
Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`:
|
||||
|
||||
```ruby
|
||||
gitlab_rails['auto_migrate'] = true
|
||||
```
|
||||
```ruby
|
||||
gitlab_rails['auto_migrate'] = true
|
||||
```
|
||||
|
||||
Save the file and reconfigure GitLab:
|
||||
Save the file and reconfigure GitLab:
|
||||
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
|
||||
1. Now that the PostgreSQL server is set up to accept remote connections, run
|
||||
`netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port
|
||||
|
@ -241,143 +243,143 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
|
|||
will be used automatically to protect your PostgreSQL traffic from
|
||||
eavesdroppers, but to protect against active ("man-in-the-middle") attackers,
|
||||
the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL
|
||||
`server.crt` file on the **primary** node by running this command:
|
||||
`server.crt` file on the **primary** node by running this command:
|
||||
|
||||
```sh
|
||||
cat ~gitlab-psql/data/server.crt
|
||||
```
|
||||
```sh
|
||||
cat ~gitlab-psql/data/server.crt
|
||||
```
|
||||
|
||||
Copy the output into a clipboard or into a local file. You
|
||||
will need it when setting up the **secondary** node! The certificate is not sensitive
|
||||
data.
|
||||
Copy the output into a clipboard or into a local file. You
|
||||
will need it when setting up the **secondary** node! The certificate is not sensitive
|
||||
data.
|
||||
|
||||
### Step 2. Configure the **secondary** server
|
||||
|
||||
1. SSH into your GitLab **secondary** server and login as root:
|
||||
|
||||
```
|
||||
sudo -i
|
||||
```
|
||||
```
|
||||
sudo -i
|
||||
```
|
||||
|
||||
1. Stop application server and Sidekiq
|
||||
|
||||
```
|
||||
gitlab-ctl stop unicorn
|
||||
gitlab-ctl stop sidekiq
|
||||
```
|
||||
```
|
||||
gitlab-ctl stop unicorn
|
||||
gitlab-ctl stop sidekiq
|
||||
```
|
||||
|
||||
NOTE: **Note**:
|
||||
This step is important so we don't try to execute anything before the node is fully configured.
|
||||
NOTE: **Note:**
|
||||
This step is important so we don't try to execute anything before the node is fully configured.
|
||||
|
||||
1. [Check TCP connectivity][rake-maintenance] to the **primary** node's PostgreSQL server:
|
||||
|
||||
```sh
|
||||
gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432]
|
||||
```
|
||||
```sh
|
||||
gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432]
|
||||
```
|
||||
|
||||
NOTE: **Note**:
|
||||
If this step fails, you may be using the wrong IP address, or a firewall may
|
||||
be preventing access to the server. Check the IP address, paying close
|
||||
attention to the difference between public and private addresses and ensure
|
||||
that, if a firewall is present, the **secondary** node is permitted to connect to the
|
||||
**primary** node on port 5432.
|
||||
NOTE: **Note:**
|
||||
If this step fails, you may be using the wrong IP address, or a firewall may
|
||||
be preventing access to the server. Check the IP address, paying close
|
||||
attention to the difference between public and private addresses and ensure
|
||||
that, if a firewall is present, the **secondary** node is permitted to connect to the
|
||||
**primary** node on port 5432.
|
||||
|
||||
1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup:
|
||||
|
||||
```
|
||||
editor server.crt
|
||||
```
|
||||
```
|
||||
editor server.crt
|
||||
```
|
||||
|
||||
1. Set up PostgreSQL TLS verification on the **secondary** node:
|
||||
|
||||
Install the `server.crt` file:
|
||||
Install the `server.crt` file:
|
||||
|
||||
```sh
|
||||
install \
|
||||
-D \
|
||||
-o gitlab-psql \
|
||||
-g gitlab-psql \
|
||||
-m 0400 \
|
||||
-T server.crt ~gitlab-psql/.postgresql/root.crt
|
||||
```
|
||||
```sh
|
||||
install \
|
||||
-D \
|
||||
-o gitlab-psql \
|
||||
-g gitlab-psql \
|
||||
-m 0400 \
|
||||
-T server.crt ~gitlab-psql/.postgresql/root.crt
|
||||
```
|
||||
|
||||
PostgreSQL will now only recognize that exact certificate when verifying TLS
|
||||
connections. The certificate can only be replicated by someone with access
|
||||
to the private key, which is **only** present on the **primary** node.
|
||||
PostgreSQL will now only recognize that exact certificate when verifying TLS
|
||||
connections. The certificate can only be replicated by someone with access
|
||||
to the private key, which is **only** present on the **primary** node.
|
||||
|
||||
1. Test that the `gitlab-psql` user can connect to the **primary** node's database
|
||||
(the default Omnibus database name is gitlabhq_production):
|
||||
|
||||
```sh
|
||||
sudo \
|
||||
-u gitlab-psql /opt/gitlab/embedded/bin/psql \
|
||||
--list \
|
||||
-U gitlab_replicator \
|
||||
-d "dbname=gitlabhq_production sslmode=verify-ca" \
|
||||
-W \
|
||||
-h <primary_node_ip>
|
||||
```
|
||||
```sh
|
||||
sudo \
|
||||
-u gitlab-psql /opt/gitlab/embedded/bin/psql \
|
||||
--list \
|
||||
-U gitlab_replicator \
|
||||
-d "dbname=gitlabhq_production sslmode=verify-ca" \
|
||||
-W \
|
||||
-h <primary_node_ip>
|
||||
```
|
||||
|
||||
When prompted enter the password you set in the first step for the
|
||||
`gitlab_replicator` user. If all worked correctly, you should see
|
||||
the list of **primary** node's databases.
|
||||
When prompted enter the password you set in the first step for the
|
||||
`gitlab_replicator` user. If all worked correctly, you should see
|
||||
the list of **primary** node's databases.
|
||||
|
||||
A failure to connect here indicates that the TLS configuration is incorrect.
|
||||
Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node
|
||||
match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node.
|
||||
A failure to connect here indicates that the TLS configuration is incorrect.
|
||||
Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node
|
||||
match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node.
|
||||
|
||||
1. Configure PostgreSQL to enable FDW support:
|
||||
|
||||
This step is similar to how we configured the **primary** instance.
|
||||
We need to enable this, to enable FDW support, even if using a single node.
|
||||
This step is similar to how we configured the **primary** instance.
|
||||
We need to enable this, to enable FDW support, even if using a single node.
|
||||
|
||||
Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
|
||||
addresses with addresses appropriate to your network configuration:
|
||||
Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
|
||||
addresses with addresses appropriate to your network configuration:
|
||||
|
||||
```ruby
|
||||
##
|
||||
## Geo Secondary role
|
||||
## - configure dependent flags automatically to enable Geo
|
||||
##
|
||||
roles ['geo_secondary_role']
|
||||
```ruby
|
||||
##
|
||||
## Geo Secondary role
|
||||
## - configure dependent flags automatically to enable Geo
|
||||
##
|
||||
roles ['geo_secondary_role']
|
||||
|
||||
##
|
||||
## Secondary address
|
||||
## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node
|
||||
##
|
||||
postgresql['listen_address'] = '<secondary_node_ip>'
|
||||
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
|
||||
##
|
||||
## Secondary address
|
||||
## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node
|
||||
##
|
||||
postgresql['listen_address'] = '<secondary_node_ip>'
|
||||
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
|
||||
|
||||
##
|
||||
## Database credentials password (defined previously in primary node)
|
||||
## - replicate same values here as defined in primary node
|
||||
##
|
||||
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
|
||||
gitlab_rails['db_password'] = '<your_password_here>'
|
||||
##
|
||||
## Database credentials password (defined previously in primary node)
|
||||
## - replicate same values here as defined in primary node
|
||||
##
|
||||
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
|
||||
gitlab_rails['db_password'] = '<your_password_here>'
|
||||
|
||||
##
|
||||
## Enable FDW support for the Geo Tracking Database (improves performance)
|
||||
##
|
||||
geo_secondary['db_fdw'] = true
|
||||
```
|
||||
##
|
||||
## Enable FDW support for the Geo Tracking Database (improves performance)
|
||||
##
|
||||
geo_secondary['db_fdw'] = true
|
||||
```
|
||||
|
||||
For external PostgreSQL instances, see [additional instructions](external_database.md).
|
||||
If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`.
|
||||
For external PostgreSQL instances, see [additional instructions](external_database.md).
|
||||
If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`.
|
||||
|
||||
1. Reconfigure GitLab for the changes to take effect:
|
||||
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
|
||||
1. Restart PostgreSQL for the IP change to take effect and reconfigure again:
|
||||
|
||||
```sh
|
||||
gitlab-ctl restart postgresql
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl restart postgresql
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
|
||||
This last reconfigure will provision the FDW configuration and enable it.
|
||||
This last reconfigure will provision the FDW configuration and enable it.
|
||||
|
||||
### Step 3. Initiate the replication process
|
||||
|
||||
|
@ -394,9 +396,9 @@ data before running `pg_basebackup`.
|
|||
|
||||
1. SSH into your GitLab **secondary** server and login as root:
|
||||
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
|
||||
1. Choose a database-friendly name to use for your **secondary** node to
|
||||
use as the replication slot name. For example, if your domain is
|
||||
|
@ -404,38 +406,40 @@ data before running `pg_basebackup`.
|
|||
name as shown in the commands below.
|
||||
|
||||
1. Execute the command below to start a backup/restore and begin the replication
|
||||
CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name.
|
||||
Using the same slot name between two secondaries will break PostgreSQL replication.
|
||||
|
||||
```sh
|
||||
gitlab-ctl replicate-geo-database \
|
||||
--slot-name=<secondary_node_name> \
|
||||
--host=<primary_node_ip>
|
||||
```
|
||||
CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name.
|
||||
Using the same slot name between two secondaries will break PostgreSQL replication.
|
||||
|
||||
When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator`
|
||||
user in the first step.
|
||||
```sh
|
||||
gitlab-ctl replicate-geo-database \
|
||||
--slot-name=<secondary_node_name> \
|
||||
--host=<primary_node_ip>
|
||||
```
|
||||
|
||||
This command also takes a number of additional options. You can use `--help`
|
||||
to list them all, but here are a couple of tips:
|
||||
- If PostgreSQL is listening on a non-standard port, add `--port=` as well.
|
||||
- If your database is too large to be transferred in 30 minutes, you will need
|
||||
to increase the timeout, e.g., `--backup-timeout=3600` if you expect the
|
||||
initial replication to take under an hour.
|
||||
- Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether
|
||||
(e.g., you know the network path is secure, or you are using a site-to-site
|
||||
VPN). This is **not** safe over the public Internet!
|
||||
- You can read more details about each `sslmode` in the
|
||||
[PostgreSQL documentation][pg-docs-ssl];
|
||||
the instructions above are carefully written to ensure protection against
|
||||
both passive eavesdroppers and active "man-in-the-middle" attackers.
|
||||
- Change the `--slot-name` to the name of the replication slot
|
||||
to be used on the **primary** database. The script will attempt to create the
|
||||
replication slot automatically if it does not exist.
|
||||
- If you're repurposing an old server into a Geo **secondary** node, you'll need to
|
||||
add `--force` to the command line.
|
||||
- When not in a production machine you can disable backup step if you
|
||||
really sure this is what you want by adding `--skip-backup`
|
||||
When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator`
|
||||
user in the first step.
|
||||
|
||||
This command also takes a number of additional options. You can use `--help`
|
||||
to list them all, but here are a couple of tips:
|
||||
|
||||
- If PostgreSQL is listening on a non-standard port, add `--port=` as well.
|
||||
- If your database is too large to be transferred in 30 minutes, you will need
|
||||
to increase the timeout, e.g., `--backup-timeout=3600` if you expect the
|
||||
initial replication to take under an hour.
|
||||
- Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether
|
||||
(e.g., you know the network path is secure, or you are using a site-to-site
|
||||
VPN). This is **not** safe over the public Internet!
|
||||
- You can read more details about each `sslmode` in the
|
||||
[PostgreSQL documentation][pg-docs-ssl];
|
||||
the instructions above are carefully written to ensure protection against
|
||||
both passive eavesdroppers and active "man-in-the-middle" attackers.
|
||||
- Change the `--slot-name` to the name of the replication slot
|
||||
to be used on the **primary** database. The script will attempt to create the
|
||||
replication slot automatically if it does not exist.
|
||||
- If you're repurposing an old server into a Geo **secondary** node, you'll need to
|
||||
add `--force` to the command line.
|
||||
- When not in a production machine you can disable backup step if you
|
||||
really sure this is what you want by adding `--skip-backup`
|
||||
|
||||
The replication process is now complete.
|
||||
|
||||
|
@ -452,42 +456,42 @@ it will need a separate read-only user to make [PostgreSQL FDW queries][FDW]
|
|||
work:
|
||||
|
||||
1. On the **primary** Geo database, enter the PostgreSQL on the console as an
|
||||
admin user. If you are using an Omnibus-managed database, log onto the **primary**
|
||||
node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production):
|
||||
admin user. If you are using an Omnibus-managed database, log onto the **primary**
|
||||
node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production):
|
||||
|
||||
```sh
|
||||
sudo \
|
||||
-u gitlab-psql /opt/gitlab/embedded/bin/psql \
|
||||
-h /var/opt/gitlab/postgresql gitlabhq_production
|
||||
```
|
||||
```sh
|
||||
sudo \
|
||||
-u gitlab-psql /opt/gitlab/embedded/bin/psql \
|
||||
-h /var/opt/gitlab/postgresql gitlabhq_production
|
||||
```
|
||||
|
||||
1. Then create the read-only user:
|
||||
|
||||
```sql
|
||||
-- NOTE: Use the password defined earlier
|
||||
CREATE USER gitlab_geo_fdw WITH password 'mypassword';
|
||||
GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw;
|
||||
GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw;
|
||||
GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw;
|
||||
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw;
|
||||
```sql
|
||||
-- NOTE: Use the password defined earlier
|
||||
CREATE USER gitlab_geo_fdw WITH password 'mypassword';
|
||||
GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw;
|
||||
GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw;
|
||||
GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw;
|
||||
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw;
|
||||
|
||||
-- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw"
|
||||
-- automatically.
|
||||
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw;
|
||||
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw;
|
||||
```
|
||||
-- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw"
|
||||
-- automatically.
|
||||
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw;
|
||||
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw;
|
||||
```
|
||||
|
||||
1. On the **secondary** nodes, change `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```
|
||||
geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw'
|
||||
```
|
||||
```
|
||||
geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw'
|
||||
```
|
||||
|
||||
1. Save the file and reconfigure GitLab for the changes to be applied:
|
||||
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
|
|
@ -4,7 +4,7 @@ This document is relevant if you are using a PostgreSQL instance that is *not
|
|||
managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or
|
||||
manually installed and configured PostgreSQL instances.
|
||||
|
||||
NOTE: **Note**:
|
||||
NOTE: **Note:**
|
||||
We strongly recommend running Omnibus-managed instances as they are actively
|
||||
developed and tested. We aim to be compatible with most external
|
||||
(not managed by Omnibus) databases but we do not guarantee compatibility.
|
||||
|
@ -13,17 +13,17 @@ developed and tested. We aim to be compatible with most external
|
|||
|
||||
1. SSH into a GitLab **primary** application server and login as root:
|
||||
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
```sh
|
||||
sudo -i
|
||||
```
|
||||
|
||||
1. Execute the command below to define the node as **primary** node:
|
||||
|
||||
```sh
|
||||
gitlab-ctl set-geo-primary-node
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl set-geo-primary-node
|
||||
```
|
||||
|
||||
This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`.
|
||||
This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`.
|
||||
|
||||
|
||||
### Configure the external database to be replicated
|
||||
|
@ -101,26 +101,27 @@ To configure the connection to the external read-replica database and enable Log
|
|||
|
||||
1. SSH into a GitLab **secondary** application server and login as root:
|
||||
|
||||
```bash
|
||||
sudo -i
|
||||
```
|
||||
```bash
|
||||
sudo -i
|
||||
```
|
||||
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the following
|
||||
|
||||
```ruby
|
||||
##
|
||||
## Geo Secondary role
|
||||
## - configure dependent flags automatically to enable Geo
|
||||
##
|
||||
roles ['geo_secondary_role']
|
||||
```ruby
|
||||
##
|
||||
## Geo Secondary role
|
||||
## - configure dependent flags automatically to enable Geo
|
||||
##
|
||||
roles ['geo_secondary_role']
|
||||
|
||||
# note this is shared between both databases,
|
||||
# make sure you define the same password in both
|
||||
gitlab_rails['db_password'] = '<your_password_here>'
|
||||
# note this is shared between both databases,
|
||||
# make sure you define the same password in both
|
||||
gitlab_rails['db_password'] = '<your_password_here>'
|
||||
|
||||
gitlab_rails['db_username'] = 'gitlab'
|
||||
gitlab_rails['db_host'] = '<database_read_replica_host>'
|
||||
```
|
||||
|
||||
gitlab_rails['db_username'] = 'gitlab'
|
||||
gitlab_rails['db_host'] = '<database_read_replica_host>'
|
||||
```
|
||||
1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure)
|
||||
|
||||
### Configure the tracking database
|
||||
|
@ -147,73 +148,72 @@ the tracking database on port 5432.
|
|||
|
||||
1. SSH into a GitLab **secondary** server and login as root:
|
||||
|
||||
```bash
|
||||
sudo -i
|
||||
```
|
||||
```bash
|
||||
sudo -i
|
||||
```
|
||||
|
||||
1. Edit `/etc/gitlab/gitlab.rb` with the connection params and credentials for
|
||||
the machine with the PostgreSQL instance:
|
||||
the machine with the PostgreSQL instance:
|
||||
|
||||
```ruby
|
||||
geo_secondary['db_username'] = 'gitlab_geo'
|
||||
geo_secondary['db_password'] = '<your_password_here>'
|
||||
```ruby
|
||||
geo_secondary['db_username'] = 'gitlab_geo'
|
||||
geo_secondary['db_password'] = '<your_password_here>'
|
||||
|
||||
geo_secondary['db_host'] = '<tracking_database_host>'
|
||||
geo_secondary['db_port'] = <tracking_database_port> # change to the correct port
|
||||
geo_secondary['db_fdw'] = true # enable FDW
|
||||
geo_postgresql['enable'] = false # don't use internal managed instance
|
||||
```
|
||||
geo_secondary['db_host'] = '<tracking_database_host>'
|
||||
geo_secondary['db_port'] = <tracking_database_port> # change to the correct port
|
||||
geo_secondary['db_fdw'] = true # enable FDW
|
||||
geo_postgresql['enable'] = false # don't use internal managed instance
|
||||
```
|
||||
|
||||
1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure)
|
||||
|
||||
1. Run the tracking database migrations:
|
||||
|
||||
```bash
|
||||
gitlab-rake geo:db:create
|
||||
gitlab-rake geo:db:migrate
|
||||
```
|
||||
```bash
|
||||
gitlab-rake geo:db:create
|
||||
gitlab-rake geo:db:migrate
|
||||
```
|
||||
|
||||
1. Configure the
|
||||
[PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html)
|
||||
connection and credentials:
|
||||
1. Configure the [PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html)
|
||||
connection and credentials:
|
||||
|
||||
Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection
|
||||
params to match your environment. Execute it to set up the FDW connection.
|
||||
Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection
|
||||
params to match your environment. Execute it to set up the FDW connection.
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
# Secondary Database connection params:
|
||||
DB_HOST="<public_ip_or_vpc_private_ip>"
|
||||
DB_NAME="gitlabhq_production"
|
||||
DB_USER="gitlab"
|
||||
DB_PASS="<your_password_here>"
|
||||
DB_PORT="5432"
|
||||
# Secondary Database connection params:
|
||||
DB_HOST="<public_ip_or_vpc_private_ip>"
|
||||
DB_NAME="gitlabhq_production"
|
||||
DB_USER="gitlab"
|
||||
DB_PASS="<your_password_here>"
|
||||
DB_PORT="5432"
|
||||
|
||||
# Tracking Database connection params:
|
||||
GEO_DB_HOST="<public_ip_or_vpc_private_ip>"
|
||||
GEO_DB_NAME="gitlabhq_geo_production"
|
||||
GEO_DB_USER="gitlab_geo"
|
||||
GEO_DB_PORT="5432"
|
||||
# Tracking Database connection params:
|
||||
GEO_DB_HOST="<public_ip_or_vpc_private_ip>"
|
||||
GEO_DB_NAME="gitlabhq_geo_production"
|
||||
GEO_DB_USER="gitlab_geo"
|
||||
GEO_DB_PORT="5432"
|
||||
|
||||
query_exec () {
|
||||
gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}"
|
||||
}
|
||||
query_exec () {
|
||||
gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}"
|
||||
}
|
||||
|
||||
query_exec "CREATE EXTENSION postgres_fdw;"
|
||||
query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');"
|
||||
query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');"
|
||||
query_exec "CREATE SCHEMA gitlab_secondary;"
|
||||
query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};"
|
||||
```
|
||||
query_exec "CREATE EXTENSION postgres_fdw;"
|
||||
query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');"
|
||||
query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');"
|
||||
query_exec "CREATE SCHEMA gitlab_secondary;"
|
||||
query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};"
|
||||
```
|
||||
|
||||
NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine,
|
||||
but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using
|
||||
`psql` for AWS RDS.
|
||||
NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine,
|
||||
but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using
|
||||
`psql` for AWS RDS.
|
||||
|
||||
1. Save the file and [restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart)
|
||||
1. Populate the FDW tables:
|
||||
|
||||
```bash
|
||||
gitlab-rake geo:db:refresh_foreign_tables
|
||||
```
|
||||
```bash
|
||||
gitlab-rake geo:db:refresh_foreign_tables
|
||||
```
|
||||
|
|
|
@ -50,17 +50,17 @@ The following steps enable a GitLab cluster to serve as the **primary** node.
|
|||
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
|
||||
|
||||
```ruby
|
||||
##
|
||||
## Enable the Geo primary role
|
||||
##
|
||||
roles ['geo_primary_role']
|
||||
```ruby
|
||||
##
|
||||
## Enable the Geo primary role
|
||||
##
|
||||
roles ['geo_primary_role']
|
||||
|
||||
##
|
||||
## Disable automatic migrations
|
||||
##
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
```
|
||||
##
|
||||
## Disable automatic migrations
|
||||
##
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
```
|
||||
|
||||
After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect.
|
||||
|
||||
|
@ -107,36 +107,36 @@ Configure the [**secondary** database](database.md) as a read-only replica of
|
|||
the **primary** database. Use the following as a guide.
|
||||
|
||||
1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the
|
||||
following:
|
||||
following:
|
||||
|
||||
```ruby
|
||||
##
|
||||
## Configure the PostgreSQL role
|
||||
##
|
||||
roles ['postgres_role']
|
||||
```ruby
|
||||
##
|
||||
## Configure the PostgreSQL role
|
||||
##
|
||||
roles ['postgres_role']
|
||||
|
||||
##
|
||||
## Secondary address
|
||||
## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node
|
||||
## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node
|
||||
##
|
||||
postgresql['listen_address'] = '<secondary_node_ip>'
|
||||
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32']
|
||||
##
|
||||
## Secondary address
|
||||
## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node
|
||||
## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node
|
||||
##
|
||||
postgresql['listen_address'] = '<secondary_node_ip>'
|
||||
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32']
|
||||
|
||||
##
|
||||
## Database credentials password (defined previously in primary node)
|
||||
## - replicate same values here as defined in primary node
|
||||
##
|
||||
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
|
||||
gitlab_rails['db_password'] = '<your_password_here>'
|
||||
##
|
||||
## Database credentials password (defined previously in primary node)
|
||||
## - replicate same values here as defined in primary node
|
||||
##
|
||||
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
|
||||
gitlab_rails['db_password'] = '<your_password_here>'
|
||||
|
||||
##
|
||||
## When running the Geo tracking database on a separate machine, disable it
|
||||
## here and allow connections from the tracking database host. And ensure
|
||||
## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above.
|
||||
##
|
||||
geo_postgresql['enable'] = false
|
||||
```
|
||||
##
|
||||
## When running the Geo tracking database on a separate machine, disable it
|
||||
## here and allow connections from the tracking database host. And ensure
|
||||
## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above.
|
||||
##
|
||||
geo_postgresql['enable'] = false
|
||||
```
|
||||
|
||||
After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect.
|
||||
|
||||
|
@ -151,47 +151,47 @@ only a single machine, rather than as a PostgreSQL cluster.
|
|||
Configure the tracking database.
|
||||
|
||||
1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the
|
||||
following:
|
||||
following:
|
||||
|
||||
```ruby
|
||||
##
|
||||
## Enable the Geo secondary tracking database
|
||||
##
|
||||
geo_postgresql['enable'] = true
|
||||
geo_postgresql['listen_address'] = '<ip_address_of_this_host>'
|
||||
geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>'
|
||||
```ruby
|
||||
##
|
||||
## Enable the Geo secondary tracking database
|
||||
##
|
||||
geo_postgresql['enable'] = true
|
||||
geo_postgresql['listen_address'] = '<ip_address_of_this_host>'
|
||||
geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>'
|
||||
|
||||
##
|
||||
## Configure FDW connection to the replica database
|
||||
##
|
||||
geo_secondary['db_fdw'] = true
|
||||
geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>'
|
||||
geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32']
|
||||
gitlab_rails['db_host'] = '<replica_database_ip>'
|
||||
##
|
||||
## Configure FDW connection to the replica database
|
||||
##
|
||||
geo_secondary['db_fdw'] = true
|
||||
geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>'
|
||||
geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32']
|
||||
gitlab_rails['db_host'] = '<replica_database_ip>'
|
||||
|
||||
# Prevent reconfigure from attempting to run migrations on the replica DB
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
# Prevent reconfigure from attempting to run migrations on the replica DB
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
|
||||
##
|
||||
## Disable all other services that aren't needed, since we don't have a role
|
||||
## that does this.
|
||||
##
|
||||
alertmanager['enable'] = false
|
||||
consul['enable'] = false
|
||||
gitaly['enable'] = false
|
||||
gitlab_monitor['enable'] = false
|
||||
gitlab_workhorse['enable'] = false
|
||||
nginx['enable'] = false
|
||||
node_exporter['enable'] = false
|
||||
pgbouncer_exporter['enable'] = false
|
||||
postgresql['enable'] = false
|
||||
prometheus['enable'] = false
|
||||
redis['enable'] = false
|
||||
redis_exporter['enable'] = false
|
||||
repmgr['enable'] = false
|
||||
sidekiq['enable'] = false
|
||||
unicorn['enable'] = false
|
||||
```
|
||||
##
|
||||
## Disable all other services that aren't needed, since we don't have a role
|
||||
## that does this.
|
||||
##
|
||||
alertmanager['enable'] = false
|
||||
consul['enable'] = false
|
||||
gitaly['enable'] = false
|
||||
gitlab_monitor['enable'] = false
|
||||
gitlab_workhorse['enable'] = false
|
||||
nginx['enable'] = false
|
||||
node_exporter['enable'] = false
|
||||
pgbouncer_exporter['enable'] = false
|
||||
postgresql['enable'] = false
|
||||
prometheus['enable'] = false
|
||||
redis['enable'] = false
|
||||
redis_exporter['enable'] = false
|
||||
repmgr['enable'] = false
|
||||
sidekiq['enable'] = false
|
||||
unicorn['enable'] = false
|
||||
```
|
||||
|
||||
After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect.
|
||||
|
||||
|
@ -211,50 +211,50 @@ following modifications:
|
|||
1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary**
|
||||
cluster, and add the following:
|
||||
|
||||
```ruby
|
||||
##
|
||||
## Enable the Geo secondary role
|
||||
##
|
||||
roles ['geo_secondary_role', 'application_role']
|
||||
```ruby
|
||||
##
|
||||
## Enable the Geo secondary role
|
||||
##
|
||||
roles ['geo_secondary_role', 'application_role']
|
||||
|
||||
##
|
||||
## Disable automatic migrations
|
||||
##
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
##
|
||||
## Disable automatic migrations
|
||||
##
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
|
||||
##
|
||||
## Configure the connection to the tracking DB. And disable application
|
||||
## servers from running tracking databases.
|
||||
##
|
||||
geo_secondary['db_host'] = '<geo_tracking_db_host>'
|
||||
geo_secondary['db_password'] = '<geo_tracking_db_password>'
|
||||
geo_postgresql['enable'] = false
|
||||
##
|
||||
## Configure the connection to the tracking DB. And disable application
|
||||
## servers from running tracking databases.
|
||||
##
|
||||
geo_secondary['db_host'] = '<geo_tracking_db_host>'
|
||||
geo_secondary['db_password'] = '<geo_tracking_db_password>'
|
||||
geo_postgresql['enable'] = false
|
||||
|
||||
##
|
||||
## Configure connection to the streaming replica database, if you haven't
|
||||
## already
|
||||
##
|
||||
gitlab_rails['db_host'] = '<replica_database_host>'
|
||||
gitlab_rails['db_password'] = '<replica_database_password>'
|
||||
##
|
||||
## Configure connection to the streaming replica database, if you haven't
|
||||
## already
|
||||
##
|
||||
gitlab_rails['db_host'] = '<replica_database_host>'
|
||||
gitlab_rails['db_password'] = '<replica_database_password>'
|
||||
|
||||
##
|
||||
## Configure connection to Redis, if you haven't already
|
||||
##
|
||||
gitlab_rails['redis_host'] = '<redis_host>'
|
||||
gitlab_rails['redis_password'] = '<redis_password>'
|
||||
##
|
||||
## Configure connection to Redis, if you haven't already
|
||||
##
|
||||
gitlab_rails['redis_host'] = '<redis_host>'
|
||||
gitlab_rails['redis_password'] = '<redis_password>'
|
||||
|
||||
##
|
||||
## If you are using custom users not managed by Omnibus, you need to specify
|
||||
## UIDs and GIDs like below, and ensure they match between servers in a
|
||||
## cluster to avoid permissions issues
|
||||
##
|
||||
user['uid'] = 9000
|
||||
user['gid'] = 9000
|
||||
web_server['uid'] = 9001
|
||||
web_server['gid'] = 9001
|
||||
registry['uid'] = 9002
|
||||
registry['gid'] = 9002
|
||||
```
|
||||
##
|
||||
## If you are using custom users not managed by Omnibus, you need to specify
|
||||
## UIDs and GIDs like below, and ensure they match between servers in a
|
||||
## cluster to avoid permissions issues
|
||||
##
|
||||
user['uid'] = 9000
|
||||
user['gid'] = 9000
|
||||
web_server['uid'] = 9001
|
||||
web_server['gid'] = 9001
|
||||
registry['uid'] = 9002
|
||||
registry['gid'] = 9002
|
||||
```
|
||||
|
||||
NOTE: **Note:**
|
||||
If you had set up PostgreSQL cluster using the omnibus package and you had set
|
||||
|
|
|
@ -80,8 +80,8 @@ In this diagram:
|
|||
- If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios.
|
||||
- A **secondary** node performs different type of synchronizations against the **primary** node, using a special
|
||||
authorization protected by JWT:
|
||||
- Repositories are cloned/updated via Git over HTTPS.
|
||||
- Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint.
|
||||
- Repositories are cloned/updated via Git over HTTPS.
|
||||
- Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint.
|
||||
|
||||
From the perspective of a user performing Git operations:
|
||||
|
||||
|
@ -107,8 +107,8 @@ The following are required to run Geo:
|
|||
- An operating system that supports OpenSSH 6.9+ (needed for
|
||||
[fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md))
|
||||
The following operating systems are known to ship with a current version of OpenSSH:
|
||||
- [CentOS](https://www.centos.org) 7.4+
|
||||
- [Ubuntu](https://www.ubuntu.com) 16.04+
|
||||
- [CentOS](https://www.centos.org) 7.4+
|
||||
- [Ubuntu](https://www.ubuntu.com) 16.04+
|
||||
- PostgreSQL 9.6+ with [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication)
|
||||
- Git 2.9+
|
||||
|
||||
|
|
|
@ -10,41 +10,42 @@ Once removed from the Geo admin page, you must stop and uninstall the **secondar
|
|||
|
||||
1. On the **secondary** node, stop GitLab:
|
||||
|
||||
```bash
|
||||
sudo gitlab-ctl stop
|
||||
```
|
||||
```bash
|
||||
sudo gitlab-ctl stop
|
||||
```
|
||||
|
||||
1. On the **secondary** node, uninstall GitLab:
|
||||
|
||||
```bash
|
||||
# Stop gitlab and remove its supervision process
|
||||
sudo gitlab-ctl uninstall
|
||||
```bash
|
||||
# Stop gitlab and remove its supervision process
|
||||
sudo gitlab-ctl uninstall
|
||||
|
||||
# Debian/Ubuntu
|
||||
sudo dpkg --remove gitlab-ee
|
||||
# Debian/Ubuntu
|
||||
sudo dpkg --remove gitlab-ee
|
||||
|
||||
# Redhat/Centos
|
||||
sudo rpm --erase gitlab-ee
|
||||
```
|
||||
# Redhat/Centos
|
||||
sudo rpm --erase gitlab-ee
|
||||
```
|
||||
|
||||
Once GitLab has been uninstalled from the **secondary** node, the replication slot must be dropped from the **primary** node's database as follows:
|
||||
|
||||
1. On the **primary** node, start a PostgreSQL console session:
|
||||
|
||||
```bash
|
||||
sudo gitlab-psql
|
||||
```
|
||||
```bash
|
||||
sudo gitlab-psql
|
||||
```
|
||||
|
||||
NOTE: **Note:**
|
||||
Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions.
|
||||
NOTE: **Note:**
|
||||
Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions.
|
||||
|
||||
1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`.
|
||||
|
||||
```sql
|
||||
SELECT * FROM pg_replication_slots;
|
||||
```
|
||||
```sql
|
||||
SELECT * FROM pg_replication_slots;
|
||||
```
|
||||
|
||||
1. Remove the replication slot for the **secondary** node:
|
||||
|
||||
```sql
|
||||
SELECT pg_drop_replication_slot('<name_of_slot>');
|
||||
```
|
||||
```sql
|
||||
SELECT pg_drop_replication_slot('<name_of_slot>');
|
||||
```
|
||||
|
|
|
@ -184,17 +184,17 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou
|
|||
|
||||
1. Start a PostgreSQL console session:
|
||||
|
||||
```sh
|
||||
sudo gitlab-psql gitlabhq_production
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-psql gitlabhq_production
|
||||
```
|
||||
|
||||
> Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions.
|
||||
Note: **Note:** Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions.
|
||||
|
||||
1. View your replication slots with:
|
||||
|
||||
```sql
|
||||
SELECT * FROM pg_replication_slots;
|
||||
```
|
||||
```sql
|
||||
SELECT * FROM pg_replication_slots;
|
||||
```
|
||||
|
||||
Slots where `active` is `f` are not active.
|
||||
|
||||
|
@ -204,9 +204,9 @@ Slots where `active` is `f` are not active.
|
|||
- If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the
|
||||
PostgreSQL console session:
|
||||
|
||||
```sql
|
||||
SELECT pg_drop_replication_slot('<name_of_extra_slot>');
|
||||
```
|
||||
```sql
|
||||
SELECT pg_drop_replication_slot('<name_of_extra_slot>');
|
||||
```
|
||||
|
||||
### Very large repositories never successfully synchronize on the **secondary** node
|
||||
|
||||
|
@ -237,82 +237,82 @@ to start again from scratch, there are a few steps that can help you:
|
|||
|
||||
1. Stop Sidekiq and the Geo LogCursor
|
||||
|
||||
It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and
|
||||
wait until the current jobs to finish processing.
|
||||
It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and
|
||||
wait until the current jobs to finish processing.
|
||||
|
||||
You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM**
|
||||
when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands.
|
||||
You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM**
|
||||
when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands.
|
||||
|
||||
```sh
|
||||
gitlab-ctl status sidekiq
|
||||
# run: sidekiq: (pid 10180) <- this is the PID you will use
|
||||
kill -TSTP 10180 # change to the correct PID
|
||||
```sh
|
||||
gitlab-ctl status sidekiq
|
||||
# run: sidekiq: (pid 10180) <- this is the PID you will use
|
||||
kill -TSTP 10180 # change to the correct PID
|
||||
|
||||
gitlab-ctl stop sidekiq
|
||||
gitlab-ctl stop geo-logcursor
|
||||
```
|
||||
gitlab-ctl stop sidekiq
|
||||
gitlab-ctl stop geo-logcursor
|
||||
```
|
||||
|
||||
You can watch sidekiq logs to know when sidekiq jobs processing have finished:
|
||||
You can watch sidekiq logs to know when sidekiq jobs processing have finished:
|
||||
|
||||
```sh
|
||||
gitlab-ctl tail sidekiq
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl tail sidekiq
|
||||
```
|
||||
|
||||
1. Rename repository storage folders and create new ones
|
||||
|
||||
```sh
|
||||
mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old
|
||||
mkdir -p /var/opt/gitlab/git-data/repositories
|
||||
chown git:git /var/opt/gitlab/git-data/repositories
|
||||
```
|
||||
```sh
|
||||
mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old
|
||||
mkdir -p /var/opt/gitlab/git-data/repositories
|
||||
chown git:git /var/opt/gitlab/git-data/repositories
|
||||
```
|
||||
|
||||
TIP: **Tip**
|
||||
You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future
|
||||
as soon as you confirmed that you don't need it anymore, to save disk space.
|
||||
TIP: **Tip**
|
||||
You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future
|
||||
as soon as you confirmed that you don't need it anymore, to save disk space.
|
||||
|
||||
1. _(Optional)_ Rename other data folders and create new ones
|
||||
|
||||
CAUTION: **Caution**:
|
||||
You may still have files on the **secondary** node that have been removed from **primary** node but
|
||||
removal have not been reflected. If you skip this step, they will never be removed
|
||||
from this Geo node.
|
||||
CAUTION: **Caution**:
|
||||
You may still have files on the **secondary** node that have been removed from **primary** node but
|
||||
removal have not been reflected. If you skip this step, they will never be removed
|
||||
from this Geo node.
|
||||
|
||||
Any uploaded content like file attachments, avatars or LFS objects are stored in a
|
||||
subfolder in one of the two paths below:
|
||||
Any uploaded content like file attachments, avatars or LFS objects are stored in a
|
||||
subfolder in one of the two paths below:
|
||||
|
||||
1. /var/opt/gitlab/gitlab-rails/shared
|
||||
1. /var/opt/gitlab/gitlab-rails/uploads
|
||||
- /var/opt/gitlab/gitlab-rails/shared
|
||||
- /var/opt/gitlab/gitlab-rails/uploads
|
||||
|
||||
To rename all of them:
|
||||
To rename all of them:
|
||||
|
||||
```sh
|
||||
gitlab-ctl stop
|
||||
```sh
|
||||
gitlab-ctl stop
|
||||
|
||||
mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old
|
||||
mkdir -p /var/opt/gitlab/gitlab-rails/shared
|
||||
mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old
|
||||
mkdir -p /var/opt/gitlab/gitlab-rails/shared
|
||||
|
||||
mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old
|
||||
mkdir -p /var/opt/gitlab/gitlab-rails/uploads
|
||||
```
|
||||
mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old
|
||||
mkdir -p /var/opt/gitlab/gitlab-rails/uploads
|
||||
```
|
||||
|
||||
Reconfigure in order to recreate the folders and make sure permissions and ownership
|
||||
are correctly
|
||||
Reconfigure in order to recreate the folders and make sure permissions and ownership
|
||||
are correctly
|
||||
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl reconfigure
|
||||
```
|
||||
|
||||
1. Reset the Tracking Database
|
||||
|
||||
```sh
|
||||
gitlab-rake geo:db:reset
|
||||
```
|
||||
```sh
|
||||
gitlab-rake geo:db:reset
|
||||
```
|
||||
|
||||
1. Restart previously stopped services
|
||||
|
||||
```sh
|
||||
gitlab-ctl start
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl start
|
||||
```
|
||||
|
||||
## Fixing Foreign Data Wrapper errors
|
||||
|
||||
|
@ -345,108 +345,106 @@ To check the configuration:
|
|||
|
||||
1. Enter the database console:
|
||||
|
||||
```sh
|
||||
gitlab-geo-psql
|
||||
```
|
||||
```sh
|
||||
gitlab-geo-psql
|
||||
```
|
||||
|
||||
1. Check whether any tables are present. If everything is working, you
|
||||
should see something like this:
|
||||
|
||||
```sql
|
||||
gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables;
|
||||
foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_n
|
||||
ame
|
||||
-------------------------+----------------------+-------------------------------------------------+-------------------------+-----------------
|
||||
----
|
||||
gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary
|
||||
gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary
|
||||
gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary
|
||||
gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary
|
||||
<snip>
|
||||
```
|
||||
```sql
|
||||
gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables;
|
||||
foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_name
|
||||
-------------------------+----------------------+-------------------------------------------------+-------------------------+---------------------
|
||||
gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary
|
||||
gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary
|
||||
gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary
|
||||
gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary
|
||||
<snip>
|
||||
```
|
||||
|
||||
However, if the query returns with `0 rows`, then continue onto the next steps.
|
||||
However, if the query returns with `0 rows`, then continue onto the next steps.
|
||||
|
||||
1. Check that the foreign server mapping is correct via `\des+`. The
|
||||
results should look something like this:
|
||||
|
||||
```sql
|
||||
gitlabhq_geo_production=# \des+
|
||||
List of foreign servers
|
||||
-[ RECORD 1 ]--------+------------------------------------------------------------
|
||||
Name | gitlab_secondary
|
||||
Owner | gitlab-psql
|
||||
Foreign-data wrapper | postgres_fdw
|
||||
Access privileges | "gitlab-psql"=U/"gitlab-psql" +
|
||||
| gitlab_geo=U/"gitlab-psql"
|
||||
Type |
|
||||
Version |
|
||||
FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production')
|
||||
Description |
|
||||
```
|
||||
```sql
|
||||
gitlabhq_geo_production=# \des+
|
||||
List of foreign servers
|
||||
-[ RECORD 1 ]--------+------------------------------------------------------------
|
||||
Name | gitlab_secondary
|
||||
Owner | gitlab-psql
|
||||
Foreign-data wrapper | postgres_fdw
|
||||
Access privileges | "gitlab-psql"=U/"gitlab-psql" +
|
||||
| gitlab_geo=U/"gitlab-psql"
|
||||
Type |
|
||||
Version |
|
||||
FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production')
|
||||
Description |
|
||||
```
|
||||
|
||||
NOTE: **Note:** Pay particular attention to the host and port under
|
||||
FDW options. That configuration should point to the Geo secondary
|
||||
database.
|
||||
NOTE: **Note:** Pay particular attention to the host and port under
|
||||
FDW options. That configuration should point to the Geo secondary
|
||||
database.
|
||||
|
||||
If you need to experiment with changing the host or password, the
|
||||
following queries demonstrate how:
|
||||
If you need to experiment with changing the host or password, the
|
||||
following queries demonstrate how:
|
||||
|
||||
```sql
|
||||
ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>');
|
||||
ALTER SERVER gitlab_secondary OPTIONS (SET port 5432);
|
||||
```
|
||||
```sql
|
||||
ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>');
|
||||
ALTER SERVER gitlab_secondary OPTIONS (SET port 5432);
|
||||
```
|
||||
|
||||
If you change the host and/or port, you will also have to adjust the
|
||||
following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl
|
||||
reconfigure`:
|
||||
If you change the host and/or port, you will also have to adjust the
|
||||
following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl
|
||||
reconfigure`:
|
||||
|
||||
- `gitlab_rails['db_host']`
|
||||
- `gitlab_rails['db_port']`
|
||||
- `gitlab_rails['db_host']`
|
||||
- `gitlab_rails['db_port']`
|
||||
|
||||
1. Check that the user mapping is configured properly via `\deu+`:
|
||||
|
||||
```sql
|
||||
gitlabhq_geo_production=# \deu+
|
||||
List of user mappings
|
||||
Server | User name | FDW Options
|
||||
------------------+------------+--------------------------------------------------------------------------------
|
||||
gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE')
|
||||
(1 row)
|
||||
```
|
||||
```sql
|
||||
gitlabhq_geo_production=# \deu+
|
||||
List of user mappings
|
||||
Server | User name | FDW Options
|
||||
------------------+------------+--------------------------------------------------------------------------------
|
||||
gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE')
|
||||
(1 row)
|
||||
```
|
||||
|
||||
Make sure the password is correct. You can test that logins work by running `psql`:
|
||||
Make sure the password is correct. You can test that logins work by running `psql`:
|
||||
|
||||
```sh
|
||||
# Connect to the tracking database as the `gitlab_geo` user
|
||||
sudo \
|
||||
-u git /opt/gitlab/embedded/bin/psql \
|
||||
-h /var/opt/gitlab/geo-postgresql \
|
||||
-p 5431 \
|
||||
-U gitlab_geo \
|
||||
-W \
|
||||
-d gitlabhq_geo_production
|
||||
```
|
||||
```sh
|
||||
# Connect to the tracking database as the `gitlab_geo` user
|
||||
sudo \
|
||||
-u git /opt/gitlab/embedded/bin/psql \
|
||||
-h /var/opt/gitlab/geo-postgresql \
|
||||
-p 5431 \
|
||||
-U gitlab_geo \
|
||||
-W \
|
||||
-d gitlabhq_geo_production
|
||||
```
|
||||
|
||||
If you need to correct the password, the following query shows how:
|
||||
If you need to correct the password, the following query shows how:
|
||||
|
||||
```sql
|
||||
ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>');
|
||||
```
|
||||
```sql
|
||||
ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>');
|
||||
```
|
||||
|
||||
If you change the user or password, you will also have to adjust the
|
||||
following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl
|
||||
reconfigure`:
|
||||
If you change the user or password, you will also have to adjust the
|
||||
following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl
|
||||
reconfigure`:
|
||||
|
||||
- `gitlab_rails['db_username']`
|
||||
- `gitlab_rails['db_password']`
|
||||
- `gitlab_rails['db_username']`
|
||||
- `gitlab_rails['db_password']`
|
||||
|
||||
If you are using [PgBouncer in front of the secondary
|
||||
database](database.md#pgbouncer-support-optional), be sure to update
|
||||
the following settings:
|
||||
If you are using [PgBouncer in front of the secondary
|
||||
database](database.md#pgbouncer-support-optional), be sure to update
|
||||
the following settings:
|
||||
|
||||
- `geo_postgresql['fdw_external_user']`
|
||||
- `geo_postgresql['fdw_external_password']`
|
||||
- `geo_postgresql['fdw_external_user']`
|
||||
- `geo_postgresql['fdw_external_password']`
|
||||
|
||||
#### Manual reload of FDW schema
|
||||
|
||||
|
@ -456,34 +454,34 @@ reload of the FDW schema. To manually reload the FDW schema:
|
|||
1. On the node running the Geo tracking database, enter the PostgreSQL console via
|
||||
the `gitlab_geo` user:
|
||||
|
||||
```sh
|
||||
sudo \
|
||||
-u git /opt/gitlab/embedded/bin/psql \
|
||||
-h /var/opt/gitlab/geo-postgresql \
|
||||
-p 5431 \
|
||||
-U gitlab_geo \
|
||||
-W \
|
||||
-d gitlabhq_geo_production
|
||||
```
|
||||
```sh
|
||||
sudo \
|
||||
-u git /opt/gitlab/embedded/bin/psql \
|
||||
-h /var/opt/gitlab/geo-postgresql \
|
||||
-p 5431 \
|
||||
-U gitlab_geo \
|
||||
-W \
|
||||
-d gitlabhq_geo_production
|
||||
```
|
||||
|
||||
Be sure to adjust the port and hostname for your configuration. You
|
||||
may be asked to enter a password.
|
||||
Be sure to adjust the port and hostname for your configuration. You
|
||||
may be asked to enter a password.
|
||||
|
||||
1. Reload the schema via:
|
||||
|
||||
```sql
|
||||
DROP SCHEMA IF EXISTS gitlab_secondary CASCADE;
|
||||
CREATE SCHEMA gitlab_secondary;
|
||||
GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo;
|
||||
IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary;
|
||||
```
|
||||
```sql
|
||||
DROP SCHEMA IF EXISTS gitlab_secondary CASCADE;
|
||||
CREATE SCHEMA gitlab_secondary;
|
||||
GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo;
|
||||
IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary;
|
||||
```
|
||||
|
||||
1. Test that queries work:
|
||||
|
||||
```sql
|
||||
SELECT * from information_schema.foreign_tables;
|
||||
SELECT * FROM gitlab_secondary.projects limit 1;
|
||||
```
|
||||
```sql
|
||||
SELECT * from information_schema.foreign_tables;
|
||||
SELECT * FROM gitlab_secondary.projects limit 1;
|
||||
```
|
||||
|
||||
[database-start-replication]: database.md#step-3-initiate-the-replication-process
|
||||
[database-pg-replication]: database.md#postgresql-replication
|
||||
|
|
|
@ -32,68 +32,68 @@ authentication method.
|
|||
|
||||
1. **[primary]** Login to your **primary** node and run:
|
||||
|
||||
```sh
|
||||
gitlab-ctl pg-password-md5 gitlab
|
||||
# Enter password: <your_password_here>
|
||||
# Confirm password: <your_password_here>
|
||||
# fca0b89a972d69f00eb3ec98a5838484
|
||||
```
|
||||
```sh
|
||||
gitlab-ctl pg-password-md5 gitlab
|
||||
# Enter password: <your_password_here>
|
||||
# Confirm password: <your_password_here>
|
||||
# fca0b89a972d69f00eb3ec98a5838484
|
||||
```
|
||||
|
||||
Copy the generated hash and edit `/etc/gitlab/gitlab.rb`:
|
||||
Copy the generated hash and edit `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```ruby
|
||||
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
|
||||
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
|
||||
```ruby
|
||||
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
|
||||
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
|
||||
|
||||
# Every node that runs Unicorn or Sidekiq needs to have the database
|
||||
# password specified as below. If you have a high-availability setup, this
|
||||
# must be present in all application nodes.
|
||||
gitlab_rails['db_password'] = '<your_password_here>'
|
||||
```
|
||||
# Every node that runs Unicorn or Sidekiq needs to have the database
|
||||
# password specified as below. If you have a high-availability setup, this
|
||||
# must be present in all application nodes.
|
||||
gitlab_rails['db_password'] = '<your_password_here>'
|
||||
```
|
||||
|
||||
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
|
||||
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
|
||||
|
||||
```ruby
|
||||
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this
|
||||
```
|
||||
```ruby
|
||||
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this
|
||||
```
|
||||
|
||||
1. **[primary]** Reconfigure and restart:
|
||||
|
||||
```sh
|
||||
sudo gitlab-ctl reconfigure
|
||||
sudo gitlab-ctl restart
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-ctl reconfigure
|
||||
sudo gitlab-ctl restart
|
||||
```
|
||||
|
||||
1. **[secondary]** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```ruby
|
||||
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
|
||||
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
|
||||
```ruby
|
||||
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
|
||||
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
|
||||
|
||||
# Every node that runs Unicorn or Sidekiq needs to have the database
|
||||
# password specified as below. If you have a high-availability setup, this
|
||||
# must be present in all application nodes.
|
||||
gitlab_rails['db_password'] = '<your_password_here>'
|
||||
# Every node that runs Unicorn or Sidekiq needs to have the database
|
||||
# password specified as below. If you have a high-availability setup, this
|
||||
# must be present in all application nodes.
|
||||
gitlab_rails['db_password'] = '<your_password_here>'
|
||||
|
||||
# Enable Foreign Data Wrapper
|
||||
geo_secondary['db_fdw'] = true
|
||||
# Enable Foreign Data Wrapper
|
||||
geo_secondary['db_fdw'] = true
|
||||
|
||||
# Secondary address in CIDR format, for example '5.6.7.8/32'
|
||||
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
|
||||
```
|
||||
# Secondary address in CIDR format, for example '5.6.7.8/32'
|
||||
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
|
||||
```
|
||||
|
||||
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
|
||||
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
|
||||
|
||||
```ruby
|
||||
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this
|
||||
```
|
||||
```ruby
|
||||
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this
|
||||
```
|
||||
|
||||
1. **[secondary]** Reconfigure and restart:
|
||||
|
||||
```sh
|
||||
sudo gitlab-ctl reconfigure
|
||||
sudo gitlab-ctl restart
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-ctl reconfigure
|
||||
sudo gitlab-ctl restart
|
||||
```
|
||||
|
||||
## Upgrading to GitLab 10.5
|
||||
|
||||
|
@ -171,9 +171,9 @@ the now-unused SSH keys from your secondaries, as they may cause problems if the
|
|||
|
||||
1. **[secondary]** Login to **all** your **secondary** nodes and run:
|
||||
|
||||
```ruby
|
||||
sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub
|
||||
```
|
||||
```ruby
|
||||
sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub
|
||||
```
|
||||
|
||||
### Hashed Storage
|
||||
|
||||
|
@ -236,12 +236,12 @@ instructions below.
|
|||
When in doubt, it does not hurt to do a resync. The easiest way to do this in
|
||||
Omnibus is the following:
|
||||
|
||||
1. Make sure you have Omnibus GitLab on the **primary** server.
|
||||
1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database.
|
||||
1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`.
|
||||
1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations.
|
||||
1. Install GitLab on the **secondary** server.
|
||||
1. Re-run the [database replication process][database-replication].
|
||||
1. Make sure you have Omnibus GitLab on the **primary** server.
|
||||
1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database.
|
||||
1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`.
|
||||
1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations.
|
||||
1. Install GitLab on the **secondary** server.
|
||||
1. Re-run the [database replication process][database-replication].
|
||||
|
||||
## Special update notes for 9.0.x
|
||||
|
||||
|
@ -262,16 +262,16 @@ is prepended with the relevant node for better clarity:
|
|||
|
||||
1. **[secondary]** Login to **all** your **secondary** nodes and stop all services:
|
||||
|
||||
```ruby
|
||||
sudo gitlab-ctl stop
|
||||
```
|
||||
```ruby
|
||||
sudo gitlab-ctl stop
|
||||
```
|
||||
|
||||
1. **[secondary]** Make a backup of the `recovery.conf` file on **all**
|
||||
**secondary** nodes to preserve PostgreSQL's credentials:
|
||||
|
||||
```sh
|
||||
sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/
|
||||
```
|
||||
```sh
|
||||
sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/
|
||||
```
|
||||
|
||||
1. **[primary]** Update the **primary** node to GitLab 9.0 following the
|
||||
[regular update docs][update]. At the end of the update, the **primary** node
|
||||
|
@ -281,136 +281,133 @@ is prepended with the relevant node for better clarity:
|
|||
stop all services except `postgresql` as we will use it to re-initialize the
|
||||
**secondary** node's database:
|
||||
|
||||
```sh
|
||||
sudo gitlab-ctl stop
|
||||
sudo gitlab-ctl start postgresql
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-ctl stop
|
||||
sudo gitlab-ctl start postgresql
|
||||
```
|
||||
|
||||
1. **[secondary]** Run the following steps on each of the **secondary** nodes:
|
||||
|
||||
1. **[secondary]** Stop all services:
|
||||
1. **[secondary]** Stop all services:
|
||||
|
||||
```sh
|
||||
sudo gitlab-ctl stop
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-ctl stop
|
||||
```
|
||||
|
||||
1. **[secondary]** Prevent running database migrations:
|
||||
1. **[secondary]** Prevent running database migrations:
|
||||
|
||||
```sh
|
||||
sudo touch /etc/gitlab/skip-auto-migrations
|
||||
```
|
||||
```sh
|
||||
sudo touch /etc/gitlab/skip-auto-migrations
|
||||
```
|
||||
|
||||
1. **[secondary]** Move the old database to another directory:
|
||||
1. **[secondary]** Move the old database to another directory:
|
||||
|
||||
```sh
|
||||
sudo mv /var/opt/gitlab/postgresql{,.bak}
|
||||
```
|
||||
```sh
|
||||
sudo mv /var/opt/gitlab/postgresql{,.bak}
|
||||
```
|
||||
|
||||
1. **[secondary]** Update to GitLab 9.0 following the [regular update docs][update].
|
||||
At the end of the update, the node will be running with PostgreSQL 9.6.
|
||||
1. **[secondary]** Update to GitLab 9.0 following the [regular update docs][update].
|
||||
At the end of the update, the node will be running with PostgreSQL 9.6.
|
||||
|
||||
1. **[secondary]** Make sure all services are up:
|
||||
1. **[secondary]** Make sure all services are up:
|
||||
|
||||
```sh
|
||||
sudo gitlab-ctl start
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-ctl start
|
||||
```
|
||||
|
||||
1. **[secondary]** Reconfigure GitLab:
|
||||
1. **[secondary]** Reconfigure GitLab:
|
||||
|
||||
```sh
|
||||
sudo gitlab-ctl reconfigure
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-ctl reconfigure
|
||||
```
|
||||
|
||||
1. **[secondary]** Run the PostgreSQL upgrade command:
|
||||
1. **[secondary]** Run the PostgreSQL upgrade command:
|
||||
|
||||
```sh
|
||||
sudo gitlab-ctl pg-upgrade
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-ctl pg-upgrade
|
||||
```
|
||||
|
||||
1. **[secondary]** See the stored credentials for the database that you will
|
||||
need to re-initialize the replication:
|
||||
1. **[secondary]** See the stored credentials for the database that you will
|
||||
need to re-initialize the replication:
|
||||
|
||||
```sh
|
||||
sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf
|
||||
```
|
||||
```sh
|
||||
sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf
|
||||
```
|
||||
|
||||
1. **[secondary]** Create the `replica.sh` script as described in the
|
||||
[database configuration document][database-source-replication].
|
||||
1. **[secondary]** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the
|
||||
embedded paths if necessary:
|
||||
|
||||
1. 1. **[secondary]** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the
|
||||
embedded paths if necessary:
|
||||
```
|
||||
#!/bin/bash
|
||||
|
||||
```
|
||||
#!/bin/bash
|
||||
PORT="5432"
|
||||
USER="gitlab_replicator"
|
||||
echo ---------------------------------------------------------------
|
||||
echo WARNING: Make sure this script is run from the secondary server
|
||||
echo ---------------------------------------------------------------
|
||||
echo
|
||||
echo Enter the IP or FQDN of the primary PostgreSQL server
|
||||
read HOST
|
||||
echo Enter the password for $USER@$HOST
|
||||
read -s PASSWORD
|
||||
echo Enter the required sslmode
|
||||
read SSLMODE
|
||||
|
||||
PORT="5432"
|
||||
USER="gitlab_replicator"
|
||||
echo ---------------------------------------------------------------
|
||||
echo WARNING: Make sure this script is run from the secondary server
|
||||
echo ---------------------------------------------------------------
|
||||
echo
|
||||
echo Enter the IP or FQDN of the primary PostgreSQL server
|
||||
read HOST
|
||||
echo Enter the password for $USER@$HOST
|
||||
read -s PASSWORD
|
||||
echo Enter the required sslmode
|
||||
read SSLMODE
|
||||
echo Stopping PostgreSQL and all GitLab services
|
||||
sudo service gitlab stop
|
||||
sudo service postgresql stop
|
||||
|
||||
echo Stopping PostgreSQL and all GitLab services
|
||||
sudo service gitlab stop
|
||||
sudo service postgresql stop
|
||||
echo Backing up postgresql.conf
|
||||
sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
|
||||
|
||||
echo Backing up postgresql.conf
|
||||
sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
|
||||
echo Cleaning up old cluster directory
|
||||
sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data
|
||||
|
||||
echo Cleaning up old cluster directory
|
||||
sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data
|
||||
echo Starting base backup as the replicator user
|
||||
echo Enter the password for $USER@$HOST
|
||||
sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
|
||||
|
||||
echo Starting base backup as the replicator user
|
||||
echo Enter the password for $USER@$HOST
|
||||
sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
|
||||
echo Writing recovery.conf file
|
||||
sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
|
||||
standby_mode = 'on'
|
||||
primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE'
|
||||
_EOF1_
|
||||
"
|
||||
|
||||
echo Writing recovery.conf file
|
||||
sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
|
||||
standby_mode = 'on'
|
||||
primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE'
|
||||
_EOF1_
|
||||
"
|
||||
echo Restoring postgresql.conf
|
||||
sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
|
||||
|
||||
echo Restoring postgresql.conf
|
||||
sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
|
||||
echo Starting PostgreSQL
|
||||
sudo service postgresql start
|
||||
```
|
||||
|
||||
echo Starting PostgreSQL
|
||||
sudo service postgresql start
|
||||
```
|
||||
1. **[secondary]** Run the recovery script using the credentials from the
|
||||
previous step:
|
||||
|
||||
1. **[secondary]** Run the recovery script using the credentials from the
|
||||
previous step:
|
||||
```sh
|
||||
sudo bash /tmp/replica.sh
|
||||
```
|
||||
|
||||
```sh
|
||||
sudo bash /tmp/replica.sh
|
||||
```
|
||||
1. **[secondary]** Reconfigure GitLab:
|
||||
|
||||
1. **[secondary]** Reconfigure GitLab:
|
||||
```sh
|
||||
sudo gitlab-ctl reconfigure
|
||||
```
|
||||
|
||||
```sh
|
||||
sudo gitlab-ctl reconfigure
|
||||
```
|
||||
1. **[secondary]** Start all services:
|
||||
|
||||
1. **[secondary]** Start all services:
|
||||
```sh
|
||||
sudo gitlab-ctl start
|
||||
```
|
||||
|
||||
```sh
|
||||
sudo gitlab-ctl start
|
||||
```
|
||||
|
||||
1. **[secondary]** Repeat the steps for the remaining **secondary** nodes.
|
||||
1. **[secondary]** Repeat the steps for the remaining **secondary** nodes.
|
||||
|
||||
1. **[primary]** After all **secondary** nodes are updated, start all services in
|
||||
**primary** node:
|
||||
|
||||
```sh
|
||||
sudo gitlab-ctl start
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-ctl start
|
||||
```
|
||||
|
||||
## Check status after updating
|
||||
|
||||
|
@ -419,9 +416,9 @@ everything is working correctly:
|
|||
|
||||
1. Run the Geo raketask on all nodes, everything should be green:
|
||||
|
||||
```sh
|
||||
sudo gitlab-rake gitlab:geo:check
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-rake gitlab:geo:check
|
||||
```
|
||||
|
||||
1. Check the **primary** node's Geo dashboard for any errors.
|
||||
1. Test the data replication by pushing code to the **primary** node and see if it
|
||||
|
@ -435,9 +432,9 @@ and it is required since 10.0.
|
|||
|
||||
1. Run database migrations on tracking database:
|
||||
|
||||
```sh
|
||||
sudo gitlab-rake geo:db:migrate
|
||||
```
|
||||
```sh
|
||||
sudo gitlab-rake geo:db:migrate
|
||||
```
|
||||
|
||||
1. Repeat this step for each **secondary** node.
|
||||
|
||||
|
|
Loading…
Reference in New Issue