kotovalexarian-likes-gitlab
/
gitlab-org--gitlab-foss
1
0
Fork 0
Code Releases Activity

Remove support for using Geo with an installation from source 

Using Geo with source-based installs was deprecated in GitLab 11.5.

This commit removes this support now that the deprecation notice has
been in place for quite a few releases.
This commit is contained in:
Douglas Barbosa Alexandre 2019-05-27 14:33:31 -03:00
parent 773de16274
commit d288846c8e
No known key found for this signature in database
GPG Key ID: 4DC4A918C347CAC9
13 changed files with 81 additions and 691 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
changelogs/unreleased/11609-geo-remove-support-for-using-geo-with-an-installation-from-source-docs.yml Normal file
View File

@ -0,0 +1,5 @@
---
title: Remove support for using Geo with an installation from source
merge_request: 28737
author:
type: other

40
doc/administration/geo/disaster_recovery/background_verification.md
View File

@ -29,12 +29,7 @@ the node more time before scheduling a planned failover.
Run the following commands in a Rails console on the **primary** node:
```sh
# Omnibus GitLab
gitlab-rails console
# Installation from source
cd /home/git/gitlab
sudo -u git -H bin/rails console RAILS_ENV=production
```
To check if automatic background verification is enabled:
@ -102,12 +97,7 @@ disable if you need. Run the following commands in a Rails console on the
**primary** node:
```sh
# Omnibus GitLab
gitlab-rails console
# Installation from source
cd /home/git/gitlab
sudo -u git -H bin/rails console RAILS_ENV=production
```
To disable automatic background re-verification:
@ -131,31 +121,15 @@ to be resynced without the backoff period:
For repositories:
- Omnibus Installation
```sh
sudo gitlab-rake geo:verification:repository:reset
```
- Source Installation
```sh
sudo -u git -H bundle exec rake geo:verification:repository:reset RAILS_ENV=production
```
```sh
sudo gitlab-rake geo:verification:repository:reset
```
For wikis:
- Omnibus Installation
```sh
sudo gitlab-rake geo:verification:wiki:reset
```
- Source Installation
```sh
sudo -u git -H bundle exec rake geo:verification:wiki:reset RAILS_ENV=production
```
```sh
sudo gitlab-rake geo:verification:wiki:reset
```
## Reconcile differences with checksum mismatches
@ -169,7 +143,7 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch
1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**:
![Project admin page](img/checksum-differences-admin-project-page.png)
1. Navigate to the project's repository directory on both **primary** and **secondary** nodes. For an installation from source, the path is usually `/home/git/repositories`. For Omnibus installs, the path is usually `/var/opt/gitlab/git-data/repositories`. Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure.
1. Navigate to the project's repository directory on both **primary** and **secondary** nodes (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure.
```sh
cd /var/opt/gitlab/git-data/repositories

8
doc/administration/geo/replication/configuration.md
View File

@ -1,9 +1,4 @@
# Geo configuration (GitLab Omnibus) **[PREMIUM ONLY]**
NOTE: **Note:**
This is the documentation for the Omnibus GitLab packages. For installations
from source, follow the [**Geo nodes configuration for installations
from source**][configuration-source] guide.
# Geo configuration **[PREMIUM ONLY]**
## Configuring a new **secondary** node
@ -303,7 +298,6 @@ See the [updating the Geo nodes document](updating_the_geo_nodes.md).
See the [troubleshooting document](troubleshooting.md).
[configuration-source]: configuration_source.md
[setup-geo-omnibus]: index.md#using-omnibus-gitlab
[Hashed Storage]: ../../repository_storage_types.md
[Disaster Recovery]: ../disaster_recovery/index.md

172
doc/administration/geo/replication/configuration_source.md
View File

@ -1,172 +0,0 @@
# Geo configuration (source) **[PREMIUM ONLY]**
NOTE: **Note:**
This documentation applies to GitLab source installations. In GitLab 11.5, this documentation was deprecated and will be removed in a future release.
Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html). For installations
using the Omnibus GitLab packages, follow the
[**Omnibus Geo nodes configuration**][configuration] guide.
## Configuring a new **secondary** node
NOTE: **Note:**
This is the final step in setting up a **secondary** node. Stages of the setup
process must be completed in the documented order. Before attempting the steps
in this stage, [complete all prior stages](index.md#using-gitlab-installed-from-source-deprecated).
The basic steps of configuring a **secondary** node are to:
- Replicate required configurations between the **primary** and **secondary** nodes.
- Configure a tracking database on each **secondary** node.
- Start GitLab on the **secondary** node.
You are encouraged to first read through all the steps before executing them
in your testing/production environment.
NOTE: **Note:**
**Do not** set up any custom authentication on **secondary** nodes, this will be handled by the **primary** node.
NOTE: **Note:**
**Do not** add anything in the **secondary** node's admin area (**Admin Area > Geo**). This is handled solely by the **primary** node.
### Step 1. Manually replicate secret GitLab values
GitLab stores a number of secret values in the `/home/git/gitlab/config/secrets.yml`
file which *must* match between the **primary** and **secondary** nodes. Until there is
a means of automatically replicating these between nodes (see [gitlab-org/gitlab-ee#3789]), they must
be manually replicated to **secondary** nodes.
1. SSH into the **primary** node, and execute the command below:
```sh
sudo cat /home/git/gitlab/config/secrets.yml
```
This will display the secrets that need to be replicated, in YAML format.
1. SSH into the **secondary** node and login as the `git` user:
```sh
sudo -i -u git
```
1. Make a backup of any existing secrets:
```sh
mv /home/git/gitlab/config/secrets.yml /home/git/gitlab/config/secrets.yml.`date +%F`
```
1. Copy `/home/git/gitlab/config/secrets.yml` from the **primary** node to the **secondary** node, or
copy-and-paste the file contents between nodes:
```sh
sudo editor /home/git/gitlab/config/secrets.yml
# paste the output of the `cat` command you ran on the primary
# save and exit
```
1. Ensure the file permissions are correct:
```sh
chown git:git /home/git/gitlab/config/secrets.yml
chmod 0600 /home/git/gitlab/config/secrets.yml
```
1. Restart GitLab
```sh
service gitlab restart
```
Once restarted, the **secondary** node will automatically start replicating missing data
from the **primary** node in a process known as backfill. Meanwhile, the **primary** node
will start to notify the **secondary** node of any changes, so that the **secondary** node can
act on those notifications immediately.
Make sure the **secondary** node is running and accessible. You can login to
the **secondary** node with the same credentials as used for the **primary** node.
### Step 2. Manually replicate the **primary** node's SSH host keys
Read [Manually replicate the **primary** node's SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys)
### Step 3. Add the **secondary** GitLab node
1. Navigate to the **primary** node's **Admin Area > Geo**
(`/admin/geo/nodes`) in your browser.
1. Add the **secondary** node by providing its full URL. **Do NOT** check the
**This is a primary node** checkbox.
1. Optionally, choose which namespaces should be replicated by the
**secondary** node. Leave blank to replicate all. Read more in
[selective synchronization](#selective-synchronization).
1. Click the **Add node** button.
1. SSH into your GitLab **secondary** server and restart the services:
```sh
service gitlab restart
```
Check if there are any common issue with your Geo setup by running:
```sh
bundle exec rake gitlab:geo:check
```
1. SSH into your GitLab **primary** server and login as root to verify the
**secondary** node is reachable or there are any common issue with your Geo setup:
```sh
bundle exec rake gitlab:geo:check
```
Once reconfigured, the **secondary** node will automatically start
replicating missing data from the **primary** node in a process known as backfill.
Meanwhile, the **primary** node will start to notify the **secondary** node of any changes, so
that the **secondary** node can act on those notifications immediately.
Make sure the **secondary** node is running and accessible.
You can log in to the **secondary** node with the same credentials as used for the **primary** node.
### Step 4. Enabling Hashed Storage
Read [Enabling Hashed Storage](configuration.md#step-4-enabling-hashed-storage).
### Step 5. (Optional) Configuring the secondary to trust the primary
You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate.
If your **primary** node is using a self-signed certificate for *HTTPS* support, you will
need to add that certificate to the **secondary** node's trust store. Retrieve the
certificate from the **primary** node and follow your distribution's instructions for
adding it to the **secondary** node's trust store. In Debian/Ubuntu, you would follow these steps:
```sh
sudo -i
cp <primary_node_certification_file> /usr/local/share/ca-certificates
update-ca-certificates
```
### Step 6. Enable Git access over HTTP/HTTPS
Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
method to be enabled. Navigate to **Admin Area > Settings**
(`/admin/application_settings`) on the **primary** node, and set
`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
### Step 7. Verify proper functioning of the secondary node
Read [Verify proper functioning of the secondary node][configuration-verify-node].
## Selective synchronization
Read [Selective synchronization][configuration-selective-replication].
## Troubleshooting
Read the [troubleshooting document][troubleshooting].
[gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789
[configuration]: configuration.md
[configuration-selective-replication]: configuration.md#selective-synchronization
[configuration-verify-node]: configuration.md#step-7-verify-proper-functioning-of-the-secondary-node
[troubleshooting]: troubleshooting.md

24
doc/administration/geo/replication/database.md
View File

@ -1,9 +1,7 @@
# Geo database replication (GitLab Omnibus) **[PREMIUM ONLY]**
# Geo database replication **[PREMIUM ONLY]**
NOTE: **Note:**
This is the documentation for the Omnibus GitLab packages. For installations
from source, follow the
[Geo database replication (source)](database_source.md) guide.
The following steps are for Omnibus installs only. Using Geo with source-based installs was **deprecated** in GitLab 11.5.
NOTE: **Note:**
If your GitLab installation uses external (not managed by Omnibus) PostgreSQL
@ -102,10 +100,15 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
else.
If you are using an external database not managed by Omnibus GitLab, you need
to create the replicator user and define a password to it manually.
For information on how to create a replication user, refer to the
[appropriate step](database_source.md#step-1-configure-the-primary-server)
in [Geo database replication (source)](database_source.md).
to create the replicator user and define a password to it manually:
```sql
--- Create a new user 'replicator'
CREATE USER gitlab_replicator;
--- Set/change a password and grants replication privilege
ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>';
```
1. Configure PostgreSQL to listen on network interfaces:
@ -340,7 +343,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
##
## Secondary address
## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node
## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node
##
postgresql['listen_address'] = '<secondary_node_ip>'
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
@ -383,8 +386,7 @@ the database on the **primary** node, replicates the database, and creates the
needed files for streaming replication.
The directories used are the defaults that are set up in Omnibus. If you have
changed any defaults or are using a source installation, configure it as you
see fit replacing the directories and paths.
changed any defaults, configure it as you see fit replacing the directories and paths.
CAUTION: **Warning:**
Make sure to run this on the **secondary** server as it removes all PostgreSQL's

439
doc/administration/geo/replication/database_source.md
View File

@ -1,439 +0,0 @@
# Geo database replication (source) **[PREMIUM ONLY]**
NOTE: **Note:**
This documentation applies to GitLab source installations. In GitLab 11.5, this documentation was deprecated and will be removed in a future release.
Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html). For installations
using the Omnibus GitLab packages, follow the
[**database replication for Omnibus GitLab**][database] guide.
NOTE: **Note:**
The stages of the setup process must be completed in the documented order.
Before attempting the steps in this stage, [complete all prior stages](index.md#using-gitlab-installed-from-source-deprecated).
This document describes the minimal steps you have to take in order to
replicate your **primary** GitLab database to a **secondary** node's database. You may
have to change some values according to your database setup, how big it is, etc.
You are encouraged to first read through all the steps before executing them
in your testing/production environment.
## PostgreSQL replication
The GitLab **primary** node where the write operations happen will connect to
**primary** database server, and the **secondary** ones which are read-only will
connect to **secondary** database servers (which are read-only too).
NOTE: **Note:**
In many databases' documentation, you will see "**primary**" being referenced as "master"
and "**secondary**" as either "slave" or "standby" server (read-only).
We recommend using [PostgreSQL replication slots][replication-slots-article]
to ensure the **primary** node retains all the data necessary for the secondaries to
recover. See below for more details.
The following guide assumes that:
- You are using PostgreSQL 9.6 or later which includes the
[`pg_basebackup` tool][pgback] and improved [Foreign Data Wrapper][FDW] support.
- You have a **primary** node already set up (the GitLab server you are
replicating from), running PostgreSQL 9.6 or later, and
you have a new **secondary** server set up with the same versions of the OS,
PostgreSQL, and GitLab on all nodes.
- The IP of the **primary** server for our examples is `198.51.100.1`, whereas the
**secondary** node's IP is `198.51.100.2`. Note that the **primary** and **secondary** servers
**must** be able to communicate over these addresses. These IP addresses can either
be public or private.
CAUTION: **Warning:**
Geo works with streaming replication. Logical replication is not supported at this time.
There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab-ee/issues/7420).
### Step 1. Configure the **primary** server
1. SSH into your GitLab **primary** server and login as root:
```sh
sudo -i
```
1. Add this node as the Geo **primary** by running:
```sh
bundle exec rake geo:set_primary_node
```
1. Create a [replication user] named `gitlab_replicator`:
```sql
--- Create a new user 'replicator'
CREATE USER gitlab_replicator;
--- Set/change a password and grants replication privilege
ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>';
```
1. Make sure your the `gitlab` database user has a password defined:
```sh
sudo \
-u postgres psql \
-d template1 \
-c "ALTER USER gitlab WITH ENCRYPTED PASSWORD '<database_password>';"
```
1. Edit the content of `database.yml` in `production:` and add the password like the example below:
```yaml
#
# PRODUCTION
#
production:
adapter: postgresql
encoding: unicode
database: gitlabhq_production
pool: 10
username: gitlab
password: <database_password>
host: /var/opt/gitlab/geo-postgresql
```
1. Set up TLS support for the PostgreSQL **primary** server:
CAUTION: **Warning**:
Only skip this step if you **know** that PostgreSQL traffic
between the **primary** and **secondary** nodes will be secured through some other
means, e.g., a known-safe physical network path or a site-to-site VPN that
you have configured.
If you are replicating your database across the open Internet, it is
**essential** that the connection is TLS-secured. Correctly configured, this
provides protection against both passive eavesdroppers and active
"man-in-the-middle" attackers.
To generate a self-signed certificate and key, run this command:
```sh
openssl req \
-nodes \
-batch \
-x509 \
-newkey rsa:4096 \
-keyout server.key \
-out server.crt \
-days 3650
```
This will create two files - `server.key` and `server.crt` - that you can
use for authentication.
Copy them to the correct location for your PostgreSQL installation:
```sh
# Copying a self-signed certificate and key
install -o postgres -g postgres -m 0400 -T server.crt ~postgres/9.x/main/data/server.crt
install -o postgres -g postgres -m 0400 -T server.key ~postgres/9.x/main/data/server.key
```
Add this configuration to `postgresql.conf`, removing any existing
configuration for `ssl_cert_file` or `ssl_key_file`:
```
ssl = on
ssl_cert_file='server.crt'
ssl_key_file='server.key'
```
1. Edit `postgresql.conf` to configure the **primary** server for streaming replication
(for Debian/Ubuntu that would be `/etc/postgresql/9.x/main/postgresql.conf`):
```
listen_address = '<primary_node_ip>'
wal_level = hot_standby
max_wal_senders = 5
min_wal_size = 80MB
max_wal_size = 1GB
max_replicaton_slots = 1 # Number of Geo secondary nodes
wal_keep_segments = 10
hot_standby = on
```
NOTE: **Note**:
Be sure to set `max_replication_slots` to the number of Geo **secondary**
nodes that you may potentially have (at least 1).
For security reasons, PostgreSQL by default only listens on the local
interface (e.g. 127.0.0.1). However, Geo needs to communicate
between the **primary** and **secondary** nodes over a common network, such as a
corporate LAN or the public Internet. For this reason, we need to
configure PostgreSQL to listen on more interfaces.
The `listen_address` option opens PostgreSQL up to external connections
with the interface corresponding to the given IP. See [the PostgreSQL
documentation][pg-docs-runtime-conn] for more details.
You may also want to edit the `wal_keep_segments` and `max_wal_senders` to
match your database replication requirements. Consult the
[PostgreSQL - Replication documentation][pg-docs-runtime-replication] for more information.
1. Set the access control on the **primary** node to allow TCP connections using the
server's public IP and set the connection from the **secondary** node to require a
password. Edit `pg_hba.conf` (for Debian/Ubuntu that would be
`/etc/postgresql/9.x/main/pg_hba.conf`):
```sh
host all all <primary_node_ip>/32 md5
host replication gitlab_replicator <secondary_node_ip>/32 md5
```
If you want to add another secondary, add one more row like the replication
one and change the IP address:
```sh
host all all <primary_node_ip>/32 md5
host replication gitlab_replicator <secondary_node_ip>/32 md5
host replication gitlab_replicator <another_secondary_node_ip>/32 md5
```
1. Restart PostgreSQL for the changes to take effect.
1. Choose a database-friendly name to use for your secondary to use as the
replication slot name. For example, if your domain is
`secondary.geo.example.com`, you may use `secondary_example` as the slot
name.
1. Create the replication slot on the **primary** node:
```sh
$ sudo -u postgres psql -c "SELECT * FROM pg_create_physical_replication_slot('secondary_example');"
slot_name | xlog_position
------------------+---------------
secondary_example |
(1 row)
```
1. Now that the PostgreSQL server is set up to accept remote connections, run
`netstat -plnt` to make sure that PostgreSQL is listening to the server's
public IP.
### Step 2. Configure the secondary server
Follow the first steps in ["configure the secondary server"][database-replication] and note that since you are installing from source, the username and
group listed as `gitlab-psql` in those steps should be replaced by `postgres`
instead. After completing the "Test that the `gitlab-psql` user can connect to
the **primary** node's database" step, continue here:
1. Edit `postgresql.conf` to configure the secondary for streaming replication
(for Debian/Ubuntu that would be `/etc/postgresql/9.*/main/postgresql.conf`):
```sh
wal_level = hot_standby
max_wal_senders = 5
checkpoint_segments = 10
wal_keep_segments = 10
hot_standby = on
```
1. Restart PostgreSQL for the changes to take effect.
#### Enable tracking database on the secondary server
Geo secondary nodes use a tracking database to keep track of replication status
and recover automatically from some replication issues. Follow the steps below to create
the tracking database.
1. On the secondary node, run the following command to create `database_geo.yml` with the
information of your secondary PostgreSQL instance:
```sh
sudo cp /home/git/gitlab/config/database_geo.yml.postgresql /home/git/gitlab/config/database_geo.yml
```
1. Edit the content of `database_geo.yml` in `production:` as in the example below:
```yaml
#
# PRODUCTION
#
production:
adapter: postgresql
encoding: unicode
database: gitlabhq_geo_production
pool: 10
username: gitlab_geo
# password:
host: /var/opt/gitlab/geo-postgresql
```
1. Create the database `gitlabhq_geo_production` on the PostgreSQL instance of the **secondary** node.
1. Set up the Geo tracking database:
```sh
bundle exec rake geo:db:migrate
```
1. Configure the [PostgreSQL FDW][FDW] connection and credentials:
Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection
params to match your environment. Execute it to set up the FDW connection.
```sh
#!/bin/bash
# Secondary Database connection params:
DB_HOST="/var/opt/gitlab/postgresql" # change to the public IP or VPC private IP if its an external server
DB_NAME="gitlabhq_production"
DB_USER="gitlab"
DB_PORT="5432"
# Tracking Database connection params:
GEO_DB_HOST="/var/opt/gitlab/geo-postgresql" # change to the public IP or VPC private IP if its an external server
GEO_DB_NAME="gitlabhq_geo_production"
GEO_DB_USER="gitlab_geo"
GEO_DB_PORT="5432"
query_exec () {
gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}"
}
query_exec "CREATE EXTENSION postgres_fdw;"
query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');"
query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}');"
query_exec "CREATE SCHEMA gitlab_secondary;"
query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};"
```
And edit the content of `database_geo.yml` and to add `fdw: true` to
the `production:` block.
### Step 3. Initiate the replication process
Below we provide a script that connects the database on the **secondary** node to
the database on the **primary** node, replicates the database, and creates the
needed files for streaming replication.
The directories used are the defaults for Debian/Ubuntu. If you have changed
any defaults, configure it as you see fit replacing the directories and paths.
CAUTION: **Warning:**
Make sure to run this on the **secondary** server as it removes all PostgreSQL's
data before running `pg_basebackup`.
1. SSH into your GitLab **secondary** server and login as root:
```sh
sudo -i
```
1. Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the
embedded paths if necessary:
```
#!/bin/bash
PORT="5432"
USER="gitlab_replicator"
echo ---------------------------------------------------------------
echo WARNING: Make sure this script is run from the secondary server
echo ---------------------------------------------------------------
echo
echo Enter the IP or FQDN of the primary PostgreSQL server
read HOST
echo Enter the password for $USER@$HOST
read -s PASSWORD
echo Enter the required sslmode
read SSLMODE
echo Stopping PostgreSQL and all GitLab services
sudo service gitlab stop
sudo service postgresql stop
echo Backing up postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
echo Cleaning up old cluster directory
sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data
echo Starting base backup as the replicator user
echo Enter the password for $USER@$HOST
sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
echo Writing recovery.conf file
sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE'
_EOF1_
"
echo Restoring postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
echo Starting PostgreSQL
sudo service postgresql start
```
1. Run it with:
```sh
bash /tmp/replica.sh
```
When prompted, enter the IP/FQDN of the **primary** node, and the password you set up
for the `gitlab_replicator` user in the first step.
You should use `verify-ca` for the `sslmode`. You can use `disable` if you
are happy to skip PostgreSQL TLS authentication altogether (e.g., you know
the network path is secure, or you are using a site-to-site VPN). This is
**not** safe over the public Internet!
You can read more details about each `sslmode` in the
[PostgreSQL documentation][pg-docs-ssl];
the instructions above are carefully written to ensure protection against
both passive eavesdroppers and active "man-in-the-middle" attackers.
The replication process is now over.
## PGBouncer support (optional)
1. First, enter the PostgreSQL console as an admin user.
1. Then create the read-only user:
```sql
-- NOTE: Use the password defined earlier
CREATE USER gitlab_geo_fdw WITH password '<your_password_here>';
GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw;
GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw;
-- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw"
-- automatically.
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw;
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw;
```
1. Enter the PostgreSQL console on the **secondary** tracking database and change the user mapping to this new user:
```
ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET user 'gitlab_geo_fdw')
```
## MySQL replication
MySQL replication is not supported for Geo.
## Troubleshooting
Read the [troubleshooting document](troubleshooting.md).
[replication-slots-article]: https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75
[pgback]: http://www.postgresql.org/docs/9.6/static/app-pgbasebackup.html
[replication user]:https://wiki.postgresql.org/wiki/Streaming_Replication
[FDW]: https://www.postgresql.org/docs/9.6/static/postgres-fdw.html
[database]: database.md
[add-geo-node]: configuration.md#step-3-add-the-secondary-gitlab-node
[database-replication]: database.md#step-2-configure-the-secondary-server
[pg-docs-ssl]: https://www.postgresql.org/docs/9.6/static/libpq-ssl.html#LIBPQ-SSL-PROTECTION
[pg-docs-runtime-conn]: https://www.postgresql.org/docs/9.6/static/runtime-config-connection.html
[pg-docs-runtime-replication]: https://www.postgresql.org/docs/9.6/static/runtime-config-replication.html

19
doc/administration/geo/replication/index.md
View File

@ -186,30 +186,13 @@ If you installed GitLab using the Omnibus packages (highly recommended):
1. Optional: [Configure a secondary LDAP server](../../auth/ldap.md) for the **secondary** node. See [notes on LDAP](#ldap).
1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md).
### Using GitLab installed from source (Deprecated)
NOTE: **Note:**
In GitLab 11.5, support for using Geo in GitLab source installations was deprecated and will be removed in a future release. Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html).
If you installed GitLab from source:
1. [Install GitLab Enterprise Edition](../../../install/installation.md) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node.
1. [Upload the GitLab License](https://docs.gitlab.com/ee/user/admin_area/license.html) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher.
1. [Set up the database replication](database_source.md) (`primary (read-write) <-> secondary (read-only)` topology).
1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). Do this step for **both** **primary** and **secondary** nodes.
1. [Configure GitLab](configuration_source.md) to set the **primary** and **secondary** nodes.
1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md).
## Post-installation documentation
After installing GitLab on the **secondary** nodes and performing the initial configuration, see the following documentation for post-installation information.
### Configuring Geo
For information on configuring Geo, see:
- [Geo configuration (GitLab Omnibus)](configuration.md).
- [Geo configuration (source)](configuration_source.md). Configuring Geo in GitLab source installations was **deprecated** in GitLab 11.5.
For information on configuring Geo, see [Geo configuration](configuration.md).
### Updating Geo

4
doc/administration/geo/replication/security_review.md
View File

@ -120,9 +120,7 @@ questions from [owasp.org](https://www.owasp.org).
### What details regarding required OS components and lock‐down needs have been defined?
- The recommended installation method (Omnibus) packages most components itself.
A from-source installation method exists. Both are documented at
<https://docs.gitlab.com/ee/administration/geo/replication/index.html>
- The supported installation method (Omnibus) packages most components itself.
- There are significant dependencies on the system-installed OpenSSH daemon (Geo
requires users to set up custom authentication methods) and the omnibus or
system-provided PostgreSQL daemon (it must be configured to listen on TCP,

2
doc/administration/geo/replication/troubleshooting.md
View File

@ -310,7 +310,7 @@ same host on different ports. That is, 5432 and 5431 respectively.
### Checking configuration
NOTE: **Note:**
The following steps are for Omnibus installs only. Using Geo with source-based installs [is deprecated](index.md#using-gitlab-installed-from-source-deprecated).
The following steps are for Omnibus installs only. Using Geo with source-based installs was **deprecated** in GitLab 11.5.
To check the configuration:

49
doc/administration/geo/replication/updating_the_geo_nodes.md
View File

@ -337,6 +337,53 @@ is prepended with the relevant node for better clarity:
1. **[secondary]** Create the `replica.sh` script as described in the
[database configuration document][database-source-replication].
1. 1. **[secondary]** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the
embedded paths if necessary:
```
#!/bin/bash
PORT="5432"
USER="gitlab_replicator"
echo ---------------------------------------------------------------
echo WARNING: Make sure this script is run from the secondary server
echo ---------------------------------------------------------------
echo
echo Enter the IP or FQDN of the primary PostgreSQL server
read HOST
echo Enter the password for $USER@$HOST
read -s PASSWORD
echo Enter the required sslmode
read SSLMODE
echo Stopping PostgreSQL and all GitLab services
sudo service gitlab stop
sudo service postgresql stop
echo Backing up postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
echo Cleaning up old cluster directory
sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data
echo Starting base backup as the replicator user
echo Enter the password for $USER@$HOST
sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
echo Writing recovery.conf file
sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE'
_EOF1_
"
echo Restoring postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
echo Starting PostgreSQL
sudo service postgresql start
```
1. **[secondary]** Run the recovery script using the credentials from the
previous step:
@ -396,8 +443,6 @@ and it is required since 10.0.
[update]: ../../../update/README.md
[database]: database.md
[database-replication]: database.md#step-3-initiate-the-replication-process
[database-source-replication]: database_source.md#step-3-initiate-the-replication-process
[Hashed Storage]: ../../repository_storage_types.md
[hashed-migration]: ../../raketasks/storage.md
[ssh-fast-lookup]: ../../operations/fast_ssh_key_lookup.md

2
doc/development/architecture.md
View File

@ -135,7 +135,7 @@ Component statuses are linked to configuration documentation for each component.
| [Runner](#gitlab-runner) | Executes GitLab CI jobs | [⤓][runner-omnibus] | [✅][runner-charts] | [⚙][runner-charts] | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#shared-runners) | [⚙][runner-source] | [⚙][runner-gdk] | CE & EE |
| [Database Migrations](#database-migrations) | Database migrations | [✅][database-migrations-omnibus] | [✅][database-migrations-charts] | [✅][database-migrations-charts] | ✅ | [⚙][database-migrations-source] | ✅ | CE & EE |
| [Certificate Management](#certificate-management) | TLS Settings, Let's Encrypt | [✅][certificate-management-omnibus] | [✅][certificate-management-charts] | [⚙][certificate-management-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#secrets-management) | [⚙][certificate-management-source] | [⚙][certificate-management-gdk] | CE & EE |
| [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | [⚙][geo-omnibus] | [❌][geo-charts] | [❌][geo-charts] | ✅ | [❌](../administration/geo/replication/configuration_source.md) | [⚙][geo-gdk] | EE Only |
| [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | [⚙][geo-omnibus] | [❌][geo-charts] | [❌][geo-charts] | ✅ | ❌ | [⚙][geo-gdk] | EE Only |
| [LDAP Authentication](#ldap-authentication) | Authenticate users against centralized LDAP directory | [⤓][ldap-omnibus] | [⤓][ldap-charts] | [⤓][ldap-charts] | [❌](https://about.gitlab.com/pricing/#gitlab-com) | [⤓][gitlab-yml] | [⤓][ldap-gdk] | CE & EE |
| [Outbound email (SMTP)](#outbound-email) | Send email messages to users | [⤓][outbound-email-omnibus] | [⤓][outbound-email-charts] | [⤓][outbound-email-charts] | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#mail-configuration) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE |
| [Inbound email (SMTP)](#inbound-email) | Receive messages to update issues | [⤓][inbound-email-omnibus] | [⤓][inbound-email-charts] | [⤓][inbound-email-charts] | [✅](https://docs.gitlab.com/ee/user/gitlab_com/#mail-configuration) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE |

4
doc/gitlab-geo/configuration_source.md
View File

@ -1,5 +1,5 @@
---
redirect_to: '../administration/geo/replication/configuration_source.md'
redirect_to: '../administration/geo/replication/configuration.md'
---
This document was moved to [another location](../administration/geo/replication/configuration_source.md).
This document was moved to [another location](../administration/geo/replication/configuration.md).

4
doc/gitlab-geo/database_source.md
View File

@ -1,5 +1,5 @@
---
redirect_to: '../administration/geo/replication/database_source.md'
redirect_to: '../administration/geo/replication/database.md'
---
This document was moved to [another location](../administration/geo/replication/database_source.md).
This document was moved to [another location](../administration/geo/replication/database.md).