Add latest changes from gitlab-org/gitlab@master
This commit is contained in:
GitLab Bot
parent
beb5d989d2
commit
1b044a566c
|
|
@ -89,7 +89,6 @@ variables:
|
|||
REVIEW_APPS_GCP_REGION: "us-central1"
|
||||
|
||||
BUILD_ASSETS_IMAGE: "true" # Set it to "false" to disable assets image building, used in `build-assets-image`
|
||||
RSPEC_FAIL_FAST_ENABLED: "true" # Set it to "false" to disable RSpec fail-fast
|
||||
SIMPLECOV: "true"
|
||||
|
||||
# For the default QA image, we use $CI_COMMIT_SHA as tag since it's always available and we override it for specific workflow.rules (see above)
|
||||
|
|
|
|||
|
|
@ -91,13 +91,6 @@
|
|||
.if-dot-com-gitlab-org-and-security-tag: &if-dot-com-gitlab-org-and-security-tag
|
||||
if: '$CI_SERVER_HOST == "gitlab.com" && $CI_PROJECT_NAMESPACE =~ /^gitlab-org($|\/security$)/ && $CI_COMMIT_TAG'
|
||||
|
||||
|
||||
.if-rspec-fail-fast-disabled: &if-rspec-fail-fast-disabled
|
||||
if: '$RSPEC_FAIL_FAST_ENABLED != "true"'
|
||||
|
||||
.if-rspec-fail-fast-skipped: &if-rspec-fail-fast-skipped
|
||||
if: '$CI_MERGE_REQUEST_LABELS =~ /pipeline:skip-rspec-fail-fast/'
|
||||
|
||||
# For Security merge requests, the gitlab-release-tools-bot triggers a new
|
||||
# pipeline for the "Pipelines for merged results" feature. If the pipeline
|
||||
# fails, we notify release managers.
|
||||
|
|
@ -1091,10 +1084,6 @@
|
|||
|
||||
.rails:rules:rspec fail-fast:
|
||||
rules:
|
||||
- <<: *if-rspec-fail-fast-disabled
|
||||
when: never
|
||||
- <<: *if-rspec-fail-fast-skipped
|
||||
when: never
|
||||
- <<: *if-not-ee
|
||||
when: never
|
||||
- <<: *if-security-merge-request
|
||||
|
|
@ -1104,10 +1093,6 @@
|
|||
|
||||
.rails:rules:fail-pipeline-early:
|
||||
rules:
|
||||
- <<: *if-rspec-fail-fast-disabled
|
||||
when: never
|
||||
- <<: *if-rspec-fail-fast-skipped
|
||||
when: never
|
||||
- <<: *if-not-ee
|
||||
when: never
|
||||
- <<: *if-security-merge-request
|
||||
|
|
|
|||
|
|
@ -1069,6 +1069,14 @@ For Omnibus GitLab installations, Praefect logs are in `/var/log/gitlab/praefect
|
|||
|
||||
GitLab also tracks [Prometheus metrics for Praefect](gitaly/#monitor-gitaly-cluster).
|
||||
|
||||
## Backup log
|
||||
|
||||
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63832) in GitLab 14.1.
|
||||
|
||||
For Omnibus installations, the backup log is located at `/var/log/gitlab/gitlab-rails/backup_json.log`.
|
||||
|
||||
This log is populated when a [GitLab backup is created](../raketasks/backup_restore.md#back-up-gitlab). You can use this log to understand how the backup process performed.
|
||||
|
||||
## Performance bar stats
|
||||
|
||||
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48149) in GitLab 13.7.
|
||||
|
|
|
|||
|
|
@ -22,10 +22,8 @@ full list of reference architectures, see
|
|||
| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` |
|
||||
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
|
||||
| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
|
||||
| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
|
||||
| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
|
||||
| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` |
|
||||
| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` |
|
||||
| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
|
||||
| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
|
||||
| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` |
|
||||
| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
|
||||
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
|
||||
|
|
@ -33,7 +31,7 @@ full list of reference architectures, see
|
|||
| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` |
|
||||
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
|
||||
| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a |
|
||||
| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
|
||||
| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
|
||||
|
||||
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
|
||||
<!-- markdownlint-disable MD029 -->
|
||||
|
|
@ -82,11 +80,6 @@ card "Database" as database {
|
|||
card "redis" as redis {
|
||||
collections "**Redis Persistent** x3" as redis_persistent #FF6347
|
||||
collections "**Redis Cache** x3" as redis_cache #FF6347
|
||||
collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347
|
||||
collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347
|
||||
|
||||
redis_persistent <.[#FF6347]- redis_persistent_sentinel
|
||||
redis_cache <.[#FF6347]- redis_cache_sentinel
|
||||
}
|
||||
|
||||
cloud "**Object Storage**" as object_storage #white
|
||||
|
|
@ -137,8 +130,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based
|
|||
|
||||
Due to better performance and availability, for data objects (such as LFS,
|
||||
uploads, or artifacts), using an [object storage service](#configure-the-object-storage)
|
||||
is recommended instead of using NFS. Using an object storage service also
|
||||
doesn't require you to provision and maintain a node.
|
||||
is recommended.
|
||||
|
||||
It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
|
||||
that to achieve full High Availability a third-party PostgreSQL database solution will be required.
|
||||
|
|
@ -169,10 +161,8 @@ To set up GitLab and its components to accommodate up to 10,000 users:
|
|||
used for shared data objects.
|
||||
1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
|
||||
more advanced code search across your entire GitLab instance.
|
||||
1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended)
|
||||
to have shared disk storage service as an alternative to Gitaly or object
|
||||
storage. You can skip this step if you're not using GitLab Pages (which
|
||||
requires NFS).
|
||||
1. [Configure NFS](#configure-nfs)
|
||||
to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage).
|
||||
|
||||
The servers start on the same 10.6.0.0/24 private network range, and can
|
||||
connect to each other freely on these addresses.
|
||||
|
|
@ -193,15 +183,9 @@ The following list includes descriptions of each server and its assigned IP:
|
|||
- `10.6.0.51`: Redis - Cache Primary
|
||||
- `10.6.0.52`: Redis - Cache Replica 1
|
||||
- `10.6.0.53`: Redis - Cache Replica 2
|
||||
- `10.6.0.71`: Sentinel - Cache 1
|
||||
- `10.6.0.72`: Sentinel - Cache 2
|
||||
- `10.6.0.73`: Sentinel - Cache 3
|
||||
- `10.6.0.61`: Redis - Queues Primary
|
||||
- `10.6.0.62`: Redis - Queues Replica 1
|
||||
- `10.6.0.63`: Redis - Queues Replica 2
|
||||
- `10.6.0.81`: Sentinel - Queues 1
|
||||
- `10.6.0.82`: Sentinel - Queues 2
|
||||
- `10.6.0.83`: Sentinel - Queues 3
|
||||
- `10.6.0.61`: Redis - Persistent Primary
|
||||
- `10.6.0.62`: Redis - Persistent Replica 1
|
||||
- `10.6.0.63`: Redis - Persistent Replica 2
|
||||
- `10.6.0.91`: Gitaly 1
|
||||
- `10.6.0.92`: Gitaly 2
|
||||
- `10.6.0.93`: Gitaly 3
|
||||
|
|
@ -792,15 +776,9 @@ to be used with GitLab. The following IPs will be used as an example:
|
|||
- `10.6.0.51`: Redis - Cache Primary
|
||||
- `10.6.0.52`: Redis - Cache Replica 1
|
||||
- `10.6.0.53`: Redis - Cache Replica 2
|
||||
- `10.6.0.71`: Sentinel - Cache 1
|
||||
- `10.6.0.72`: Sentinel - Cache 2
|
||||
- `10.6.0.73`: Sentinel - Cache 3
|
||||
- `10.6.0.61`: Redis - Queues Primary
|
||||
- `10.6.0.62`: Redis - Queues Replica 1
|
||||
- `10.6.0.63`: Redis - Queues Replica 2
|
||||
- `10.6.0.81`: Sentinel - Queues 1
|
||||
- `10.6.0.82`: Sentinel - Queues 2
|
||||
- `10.6.0.83`: Sentinel - Queues 3
|
||||
- `10.6.0.61`: Redis - Persistent Primary
|
||||
- `10.6.0.62`: Redis - Persistent Replica 1
|
||||
- `10.6.0.63`: Redis - Persistent Replica 2
|
||||
|
||||
### Providing your own Redis instance
|
||||
|
||||
|
|
@ -812,7 +790,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p
|
|||
Note the Redis node's IP address or hostname, port, and password (if required).
|
||||
These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails).
|
||||
|
||||
### Configure the Redis and Sentinel Cache cluster
|
||||
### Configure the Redis Cache cluster
|
||||
|
||||
This is the section where we install and set up the new Redis Cache instances.
|
||||
|
||||
|
|
@ -830,8 +808,12 @@ a node and change its status from primary to replica (and vice versa).
|
|||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
# Specify server role as 'redis_master_role' and enable Consul agent
|
||||
roles(['redis_master_role', 'consul_role'])
|
||||
# Specify server roles as 'redis_master_role' with sentinel and the Consul agent
|
||||
roles ['redis_sentinel_role', 'redis_master_role', 'consul_role']
|
||||
|
||||
# Set IP bind address and Quorum number for Redis Sentinel service
|
||||
sentinel['bind'] = '0.0.0.0'
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach to.
|
||||
# You can also set bind to '0.0.0.0' which listen in all interfaces.
|
||||
|
|
@ -843,8 +825,19 @@ a node and change its status from primary to replica (and vice versa).
|
|||
# machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis (use the same password in all nodes).
|
||||
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis and replicas (use the same password in all nodes).
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
|
||||
## Must be the same in every Redis node
|
||||
redis['master_name'] = 'gitlab-redis-cache'
|
||||
|
||||
## The IP of this primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.51'
|
||||
|
||||
# Set the Redis Cache instance as an LRU
|
||||
# 90% of available RAM in MB
|
||||
|
|
@ -878,10 +871,6 @@ a node and change its status from primary to replica (and vice versa).
|
|||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
You can specify multiple roles, like sentinel and Redis, as:
|
||||
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
|
||||
[roles](https://docs.gitlab.com/omnibus/roles/).
|
||||
|
||||
#### Configure the replica Redis Cache nodes
|
||||
|
||||
1. SSH in to the **replica** Redis server.
|
||||
|
|
@ -889,11 +878,15 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
|
||||
on the page, and to select the correct Omnibus GitLab package, with the same version
|
||||
and type (Community or Enterprise editions) as your current install.
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`:
|
||||
|
||||
```ruby
|
||||
# Specify server role as 'redis_replica_role' and enable Consul agent
|
||||
roles(['redis_replica_role', 'consul_role'])
|
||||
# Specify server roles as 'redis_sentinel_role' and 'redis_replica_role'
|
||||
roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role']
|
||||
|
||||
# Set IP bind address and Quorum number for Redis Sentinel service
|
||||
sentinel['bind'] = '0.0.0.0'
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach to.
|
||||
# You can also set bind to '0.0.0.0' which listen in all interfaces.
|
||||
|
|
@ -905,16 +898,20 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
# machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
# The same password for Redis authentication you set up for the primary node.
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
|
||||
# The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.51'
|
||||
|
||||
# Port of primary Redis server, uncomment to change to non default. Defaults
|
||||
# to `6379`.
|
||||
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis and replicas (use the same password in all nodes).
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
|
||||
## Must be the same in every Redis node
|
||||
redis['master_name'] = 'gitlab-redis-cache'
|
||||
|
||||
## The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.51'
|
||||
|
||||
# Set the Redis Cache instance as an LRU
|
||||
# 90% of available RAM in MB
|
||||
redis['maxmemory'] = '13500mb'
|
||||
|
|
@ -949,15 +946,6 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
1. Go through the steps again for all the other replica nodes, and
|
||||
make sure to set up the IPs correctly.
|
||||
|
||||
You can specify multiple roles, like sentinel and Redis, as:
|
||||
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
|
||||
[roles](https://docs.gitlab.com/omnibus/roles/).
|
||||
|
||||
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
|
||||
a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a
|
||||
`gitlab-ctl reconfigure`, they will get their configuration restored by
|
||||
the same Sentinels.
|
||||
|
||||
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
|
||||
are supported and can be added if needed.
|
||||
|
||||
|
|
@ -967,133 +955,15 @@ are supported and can be added if needed.
|
|||
</a>
|
||||
</div>
|
||||
|
||||
#### Configure the Sentinel Cache nodes
|
||||
### Configure the Redis Persistent cluster
|
||||
|
||||
Now that the Redis servers are all set up, let's configure the Sentinel
|
||||
servers. The following IPs will be used as an example:
|
||||
|
||||
- `10.6.0.71`: Sentinel - Cache 1
|
||||
- `10.6.0.72`: Sentinel - Cache 2
|
||||
- `10.6.0.73`: Sentinel - Cache 3
|
||||
|
||||
NOTE:
|
||||
If you're using an external Redis Sentinel instance, be sure to exclude the
|
||||
`requirepass` parameter from the Sentinel configuration. This parameter causes
|
||||
clients to report `NOAUTH Authentication required.`.
|
||||
[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279).
|
||||
|
||||
To configure the Sentinel Cache server:
|
||||
|
||||
1. SSH in to the server that will host Consul/Sentinel.
|
||||
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
|
||||
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
|
||||
on the page, and to select the correct Omnibus GitLab package, with the same version
|
||||
and type (Community or Enterprise editions) as your current install.
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
roles(['redis_sentinel_role', 'consul_role'])
|
||||
|
||||
## Must be the same in every sentinel node
|
||||
redis['master_name'] = 'gitlab-redis-cache'
|
||||
|
||||
## The same password for Redis authentication you set up for the primary node.
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
|
||||
## The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.51'
|
||||
|
||||
## Define a port so Redis can listen for TCP requests which will allow other
|
||||
## machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
## Port of primary Redis server, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
## Configure Sentinel's IP
|
||||
sentinel['bind'] = '10.6.0.71'
|
||||
|
||||
## Port that Sentinel listens on, uncomment to change to non default. Defaults
|
||||
## to `26379`.
|
||||
#sentinel['port'] = 26379
|
||||
|
||||
## Quorum must reflect the amount of voting sentinels it take to start a failover.
|
||||
## Value must NOT be greater then the amount of sentinels.
|
||||
##
|
||||
## The quorum can be used to tune Sentinel in two ways:
|
||||
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
|
||||
## we deploy, we are basically making Sentinel more sensible to primary failures,
|
||||
## triggering a failover as soon as even just a minority of Sentinels is no longer
|
||||
## able to talk with the primary.
|
||||
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
|
||||
## making Sentinel able to failover only when there are a very large number (larger
|
||||
## than majority) of well connected Sentinels which agree about the primary being down.s
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
## Consider unresponsive server down after x amount of ms.
|
||||
#sentinel['down_after_milliseconds'] = 10000
|
||||
|
||||
## Specifies the failover timeout in milliseconds. It is used in many ways:
|
||||
##
|
||||
## - The time needed to re-start a failover after a previous failover was
|
||||
## already tried against the same primary by a given Sentinel, is two
|
||||
## times the failover timeout.
|
||||
##
|
||||
## - The time needed for a replica replicating to a wrong primary according
|
||||
## to a Sentinel current configuration, to be forced to replicate
|
||||
## with the right primary, is exactly the failover timeout (counting since
|
||||
## the moment a Sentinel detected the misconfiguration).
|
||||
##
|
||||
## - The time needed to cancel a failover that is already in progress but
|
||||
## did not produced any configuration change (REPLICAOF NO ONE yet not
|
||||
## acknowledged by the promoted replica).
|
||||
##
|
||||
## - The maximum time a failover in progress waits for all the replica to be
|
||||
## reconfigured as replicas of the new primary. However even after this time
|
||||
## the replicas will be reconfigured by the Sentinels anyway, but not with
|
||||
## the exact parallel-syncs progression as specified.
|
||||
#sentinel['failover_timeout'] = 60000
|
||||
|
||||
## Enable service discovery for Prometheus
|
||||
consul['monitoring_service_discovery'] = true
|
||||
|
||||
## The IPs of the Consul server nodes
|
||||
## You can also use FQDNs and intermix them with IPs
|
||||
consul['configuration'] = {
|
||||
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
|
||||
}
|
||||
|
||||
# Set the network addresses that the exporters will listen on
|
||||
node_exporter['listen_address'] = '0.0.0.0:9100'
|
||||
redis_exporter['listen_address'] = '0.0.0.0:9121'
|
||||
|
||||
# Prevent database migrations from running on upgrade automatically
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
```
|
||||
|
||||
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
|
||||
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
|
||||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
1. Go through the steps again for all the other Consul/Sentinel nodes, and
|
||||
make sure you set up the correct IPs.
|
||||
|
||||
<div align="right">
|
||||
<a type="button" class="btn btn-default" href="#setup-components">
|
||||
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
### Configure the Redis and Sentinel Queues cluster
|
||||
|
||||
This is the section where we install and set up the new Redis Queues instances.
|
||||
This is the section where we install and set up the new Redis Persistent instances.
|
||||
|
||||
Both the primary and replica Redis nodes need the same password defined in
|
||||
`redis['password']`. At any time during a failover, the Sentinels can reconfigure
|
||||
a node and change its status from primary to replica (and vice versa).
|
||||
|
||||
#### Configure the primary Redis Queues node
|
||||
#### Configure the primary Redis Persistent node
|
||||
|
||||
1. SSH in to the **Primary** Redis server.
|
||||
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
|
||||
|
|
@ -1103,8 +973,12 @@ a node and change its status from primary to replica (and vice versa).
|
|||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
# Specify server role as 'redis_master_role' and enable Consul agent
|
||||
roles(['redis_master_role', 'consul_role'])
|
||||
# Specify server roles as 'redis_master_role' with Sentinel and the Consul agent
|
||||
roles ['redis_sentinel_role', 'redis_master_role', 'consul_role']
|
||||
|
||||
# Set IP bind address and Quorum number for Redis Sentinel service
|
||||
sentinel['bind'] = '0.0.0.0'
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach to.
|
||||
# You can also set bind to '0.0.0.0' which listen in all interfaces.
|
||||
|
|
@ -1116,8 +990,19 @@ a node and change its status from primary to replica (and vice versa).
|
|||
# machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis (use the same password in all nodes).
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis and replicas (use the same password in all nodes).
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
|
||||
## Must be the same in every Redis node
|
||||
redis['master_name'] = 'gitlab-redis-persistent'
|
||||
|
||||
## The IP of this primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.61'
|
||||
|
||||
## Enable service discovery for Prometheus
|
||||
consul['monitoring_service_discovery'] = true
|
||||
|
|
@ -1141,13 +1026,9 @@ a node and change its status from primary to replica (and vice versa).
|
|||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
You can specify multiple roles, like sentinel and Redis, as:
|
||||
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
|
||||
[roles](https://docs.gitlab.com/omnibus/roles/).
|
||||
#### Configure the replica Redis Persistent nodes
|
||||
|
||||
#### Configure the replica Redis Queues nodes
|
||||
|
||||
1. SSH in to the **replica** Redis Queue server.
|
||||
1. SSH in to the **replica** Redis Persistent server.
|
||||
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
|
||||
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
|
||||
on the page, and to select the correct Omnibus GitLab package, with the same version
|
||||
|
|
@ -1155,8 +1036,12 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
# Specify server role as 'redis_replica_role' and enable Consul agent
|
||||
roles(['redis_replica_role', 'consul_role'])
|
||||
# Specify server roles as 'redis_sentinel_role' and 'redis_replica_role'
|
||||
roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role']
|
||||
|
||||
# Set IP bind address and Quorum number for Redis Sentinel service
|
||||
sentinel['bind'] = '0.0.0.0'
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach to.
|
||||
# You can also set bind to '0.0.0.0' which listen in all interfaces.
|
||||
|
|
@ -1168,16 +1053,20 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
# machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
# The same password for Redis authentication you set up for the primary node.
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
|
||||
## Must be the same in every Redis node
|
||||
redis['master_name'] = 'gitlab-redis-persistent'
|
||||
|
||||
# The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.61'
|
||||
|
||||
# Port of primary Redis server, uncomment to change to non default. Defaults
|
||||
# to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
## Enable service discovery for Prometheus
|
||||
consul['monitoring_service_discovery'] = true
|
||||
|
||||
|
|
@ -1202,15 +1091,6 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
1. Go through the steps again for all the other replica nodes, and
|
||||
make sure to set up the IPs correctly.
|
||||
|
||||
You can specify multiple roles, like sentinel and Redis, as:
|
||||
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
|
||||
[roles](https://docs.gitlab.com/omnibus/roles/).
|
||||
|
||||
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
|
||||
a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a
|
||||
`gitlab-ctl reconfigure`, they will get their configuration restored by
|
||||
the same Sentinels.
|
||||
|
||||
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
|
||||
are supported and can be added if needed.
|
||||
|
||||
|
|
@ -1220,124 +1100,6 @@ are supported and can be added if needed.
|
|||
</a>
|
||||
</div>
|
||||
|
||||
#### Configure the Sentinel Queues nodes
|
||||
|
||||
Now that the Redis servers are all set up, let's configure the Sentinel
|
||||
servers. The following IPs will be used as an example:
|
||||
|
||||
- `10.6.0.81`: Sentinel - Queues 1
|
||||
- `10.6.0.82`: Sentinel - Queues 2
|
||||
- `10.6.0.83`: Sentinel - Queues 3
|
||||
|
||||
NOTE:
|
||||
If you're using an external Redis Sentinel instance, be sure to exclude the
|
||||
`requirepass` parameter from the Sentinel configuration. This parameter causes
|
||||
clients to report `NOAUTH Authentication required.`.
|
||||
[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279).
|
||||
|
||||
To configure the Sentinel Queues server:
|
||||
|
||||
1. SSH in to the server that will host Sentinel.
|
||||
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
|
||||
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
|
||||
on the page, and to select the correct Omnibus GitLab package, with the same version
|
||||
and type (Community or Enterprise editions) as your current install.
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
roles(['redis_sentinel_role', 'consul_role'])
|
||||
|
||||
## Must be the same in every sentinel node
|
||||
redis['master_name'] = 'gitlab-redis-persistent'
|
||||
|
||||
## The same password for Redis authentication you set up for the primary node.
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
|
||||
## The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.61'
|
||||
|
||||
## Define a port so Redis can listen for TCP requests which will allow other
|
||||
## machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
## Port of primary Redis server, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
## Configure Sentinel's IP
|
||||
sentinel['bind'] = '10.6.0.81'
|
||||
|
||||
## Port that Sentinel listens on, uncomment to change to non default. Defaults
|
||||
## to `26379`.
|
||||
#sentinel['port'] = 26379
|
||||
|
||||
## Quorum must reflect the amount of voting sentinels it take to start a failover.
|
||||
## Value must NOT be greater then the amount of sentinels.
|
||||
##
|
||||
## The quorum can be used to tune Sentinel in two ways:
|
||||
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
|
||||
## we deploy, we are basically making Sentinel more sensible to primary failures,
|
||||
## triggering a failover as soon as even just a minority of Sentinels is no longer
|
||||
## able to talk with the primary.
|
||||
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
|
||||
## making Sentinel able to failover only when there are a very large number (larger
|
||||
## than majority) of well connected Sentinels which agree about the primary being down.s
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
## Consider unresponsive server down after x amount of ms.
|
||||
#sentinel['down_after_milliseconds'] = 10000
|
||||
|
||||
## Specifies the failover timeout in milliseconds. It is used in many ways:
|
||||
##
|
||||
## - The time needed to re-start a failover after a previous failover was
|
||||
## already tried against the same primary by a given Sentinel, is two
|
||||
## times the failover timeout.
|
||||
##
|
||||
## - The time needed for a replica replicating to a wrong primary according
|
||||
## to a Sentinel current configuration, to be forced to replicate
|
||||
## with the right primary, is exactly the failover timeout (counting since
|
||||
## the moment a Sentinel detected the misconfiguration).
|
||||
##
|
||||
## - The time needed to cancel a failover that is already in progress but
|
||||
## did not produced any configuration change (REPLICAOF NO ONE yet not
|
||||
## acknowledged by the promoted replica).
|
||||
##
|
||||
## - The maximum time a failover in progress waits for all the replica to be
|
||||
## reconfigured as replicas of the new primary. However even after this time
|
||||
## the replicas will be reconfigured by the Sentinels anyway, but not with
|
||||
## the exact parallel-syncs progression as specified.
|
||||
#sentinel['failover_timeout'] = 60000
|
||||
|
||||
## Enable service discovery for Prometheus
|
||||
consul['monitoring_service_discovery'] = true
|
||||
|
||||
## The IPs of the Consul server nodes
|
||||
## You can also use FQDNs and intermix them with IPs
|
||||
consul['configuration'] = {
|
||||
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
|
||||
}
|
||||
|
||||
# Set the network addresses that the exporters will listen on
|
||||
node_exporter['listen_address'] = '0.0.0.0:9100'
|
||||
redis_exporter['listen_address'] = '0.0.0.0:9121'
|
||||
|
||||
# Prevent database migrations from running on upgrade automatically
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
```
|
||||
|
||||
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
|
||||
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
|
||||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
1. Go through the steps again for all the other Sentinel nodes, and
|
||||
make sure you set up the correct IPs.
|
||||
|
||||
<div align="right">
|
||||
<a type="button" class="btn btn-default" href="#setup-components">
|
||||
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
## Configure Gitaly Cluster
|
||||
|
||||
[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories.
|
||||
|
|
@ -1857,30 +1619,30 @@ To configure the Sidekiq nodes, on each one:
|
|||
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
|
||||
|
||||
gitlab_rails['redis_cache_sentinels'] = [
|
||||
{host: '10.6.0.71', port: 26379},
|
||||
{host: '10.6.0.72', port: 26379},
|
||||
{host: '10.6.0.73', port: 26379},
|
||||
{host: '10.6.0.51', port: 26379},
|
||||
{host: '10.6.0.52', port: 26379},
|
||||
{host: '10.6.0.53', port: 26379},
|
||||
]
|
||||
|
||||
## Second cluster that will host the queues, shared state, and actioncable
|
||||
## Second cluster that will host the persistent queues, shared state, and actioncable
|
||||
gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
|
||||
gitlab_rails['redis_queues_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
gitlab_rails['redis_shared_state_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
gitlab_rails['redis_actioncable_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
|
||||
# Gitaly Cluster
|
||||
|
|
@ -2032,30 +1794,30 @@ On each node perform the following:
|
|||
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
|
||||
|
||||
gitlab_rails['redis_cache_sentinels'] = [
|
||||
{host: '10.6.0.71', port: 26379},
|
||||
{host: '10.6.0.72', port: 26379},
|
||||
{host: '10.6.0.73', port: 26379},
|
||||
{host: '10.6.0.51', port: 26379},
|
||||
{host: '10.6.0.52', port: 26379},
|
||||
{host: '10.6.0.53', port: 26379},
|
||||
]
|
||||
|
||||
## Second cluster that will host the queues, shared state, and actionable
|
||||
## Second cluster that will host the persistent queues, shared state, and actionable
|
||||
gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
|
||||
gitlab_rails['redis_queues_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
gitlab_rails['redis_shared_state_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
gitlab_rails['redis_actioncable_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
|
||||
# Set the network addresses that the exporters used for monitoring will listen on
|
||||
|
|
@ -2127,7 +1889,7 @@ On each node perform the following:
|
|||
|
||||
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
1. If you're [using NFS](#configure-nfs-optional):
|
||||
1. If you're [using NFS](#configure-nfs):
|
||||
1. If necessary, install the NFS client utility packages using the following
|
||||
commands:
|
||||
|
||||
|
|
@ -2271,7 +2033,7 @@ To configure the Monitoring node:
|
|||
## Configure the object storage
|
||||
|
||||
GitLab supports using an object storage service for holding numerous types of data.
|
||||
It's recommended over [NFS](#configure-nfs-optional) and in general it's better
|
||||
It's recommended over [NFS](#configure-nfs) and in general it's better
|
||||
in larger setups as object storage is typically much more performant, reliable,
|
||||
and scalable.
|
||||
|
||||
|
|
@ -2341,7 +2103,7 @@ cluster alongside your instance, read how to
|
|||
</a>
|
||||
</div>
|
||||
|
||||
## Configure NFS (optional)
|
||||
## Configure NFS
|
||||
|
||||
[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly)
|
||||
are recommended over NFS wherever possible for improved performance. If you intend
|
||||
|
|
@ -2415,10 +2177,8 @@ services where applicable):
|
|||
| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` |
|
||||
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
|
||||
| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
|
||||
| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
|
||||
| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
|
||||
| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` |
|
||||
| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` |
|
||||
| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
|
||||
| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
|
||||
| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` |
|
||||
| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
|
||||
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
|
||||
|
|
@ -2474,11 +2234,6 @@ card "Database" as database {
|
|||
card "redis" as redis {
|
||||
collections "**Redis Persistent** x3" as redis_persistent #FF6347
|
||||
collections "**Redis Cache** x3" as redis_cache #FF6347
|
||||
collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347
|
||||
collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347
|
||||
|
||||
redis_persistent <.[#FF6347]- redis_persistent_sentinel
|
||||
redis_cache <.[#FF6347]- redis_cache_sentinel
|
||||
}
|
||||
|
||||
cloud "**Object Storage**" as object_storage #white
|
||||
|
|
|
|||
|
|
@ -22,10 +22,8 @@ full list of reference architectures, see
|
|||
| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` |
|
||||
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
|
||||
| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` |
|
||||
| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
|
||||
| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
|
||||
| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` |
|
||||
| Redis Sentinel - Queues / Shared State<sup>2</sup>| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` |
|
||||
| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
|
||||
| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.large` | `D4s v3` |
|
||||
| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` |
|
||||
| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
|
||||
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
|
||||
|
|
@ -33,7 +31,7 @@ full list of reference architectures, see
|
|||
| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` |
|
||||
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
|
||||
| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a |
|
||||
| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
|
||||
| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
|
||||
|
||||
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
|
||||
<!-- markdownlint-disable MD029 -->
|
||||
|
|
@ -82,11 +80,6 @@ card "Database" as database {
|
|||
card "redis" as redis {
|
||||
collections "**Redis Persistent** x3" as redis_persistent #FF6347
|
||||
collections "**Redis Cache** x3" as redis_cache #FF6347
|
||||
collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347
|
||||
collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347
|
||||
|
||||
redis_persistent <.[#FF6347]- redis_persistent_sentinel
|
||||
redis_cache <.[#FF6347]- redis_cache_sentinel
|
||||
}
|
||||
|
||||
cloud "**Object Storage**" as object_storage #white
|
||||
|
|
@ -137,8 +130,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based
|
|||
|
||||
Due to better performance and availability, for data objects (such as LFS,
|
||||
uploads, or artifacts), using an [object storage service](#configure-the-object-storage)
|
||||
is recommended instead of using NFS. Using an object storage service also
|
||||
doesn't require you to provision and maintain a node.
|
||||
is recommended.
|
||||
|
||||
It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
|
||||
that to achieve full High Availability a third-party PostgreSQL database solution will be required.
|
||||
|
|
@ -169,10 +161,9 @@ To set up GitLab and its components to accommodate up to 25,000 users:
|
|||
used for shared data objects.
|
||||
1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
|
||||
more advanced code search across your entire GitLab instance.
|
||||
1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended)
|
||||
to have shared disk storage service as an alternative to Gitaly or object
|
||||
storage. You can skip this step if you're not using GitLab Pages (which
|
||||
requires NFS).
|
||||
1. [Configure NFS](#configure-nfs)
|
||||
to have shared disk storage service for certain GitLab operations (non
|
||||
Gitaly or Object Storage).
|
||||
|
||||
The servers start on the same 10.6.0.0/24 private network range, and can
|
||||
connect to each other freely on these addresses.
|
||||
|
|
@ -193,15 +184,9 @@ The following list includes descriptions of each server and its assigned IP:
|
|||
- `10.6.0.51`: Redis - Cache Primary
|
||||
- `10.6.0.52`: Redis - Cache Replica 1
|
||||
- `10.6.0.53`: Redis - Cache Replica 2
|
||||
- `10.6.0.71`: Sentinel - Cache 1
|
||||
- `10.6.0.72`: Sentinel - Cache 2
|
||||
- `10.6.0.73`: Sentinel - Cache 3
|
||||
- `10.6.0.61`: Redis - Queues Primary
|
||||
- `10.6.0.62`: Redis - Queues Replica 1
|
||||
- `10.6.0.63`: Redis - Queues Replica 2
|
||||
- `10.6.0.81`: Sentinel - Queues 1
|
||||
- `10.6.0.82`: Sentinel - Queues 2
|
||||
- `10.6.0.83`: Sentinel - Queues 3
|
||||
- `10.6.0.61`: Redis - Persistent Primary
|
||||
- `10.6.0.62`: Redis - Persistent Replica 1
|
||||
- `10.6.0.63`: Redis - Persistent Replica 2
|
||||
- `10.6.0.91`: Gitaly 1
|
||||
- `10.6.0.92`: Gitaly 2
|
||||
- `10.6.0.93`: Gitaly 3
|
||||
|
|
@ -794,15 +779,9 @@ to be used with GitLab. The following IPs will be used as an example:
|
|||
- `10.6.0.51`: Redis - Cache Primary
|
||||
- `10.6.0.52`: Redis - Cache Replica 1
|
||||
- `10.6.0.53`: Redis - Cache Replica 2
|
||||
- `10.6.0.71`: Sentinel - Cache 1
|
||||
- `10.6.0.72`: Sentinel - Cache 2
|
||||
- `10.6.0.73`: Sentinel - Cache 3
|
||||
- `10.6.0.61`: Redis - Queues Primary
|
||||
- `10.6.0.62`: Redis - Queues Replica 1
|
||||
- `10.6.0.63`: Redis - Queues Replica 2
|
||||
- `10.6.0.81`: Sentinel - Queues 1
|
||||
- `10.6.0.82`: Sentinel - Queues 2
|
||||
- `10.6.0.83`: Sentinel - Queues 3
|
||||
- `10.6.0.61`: Redis - Persistent Primary
|
||||
- `10.6.0.62`: Redis - Persistent Replica 1
|
||||
- `10.6.0.63`: Redis - Persistent Replica 2
|
||||
|
||||
### Providing your own Redis instance
|
||||
|
||||
|
|
@ -814,7 +793,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p
|
|||
Note the Redis node's IP address or hostname, port, and password (if required).
|
||||
These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails).
|
||||
|
||||
### Configure the Redis and Sentinel Cache cluster
|
||||
### Configure the Redis Cache cluster
|
||||
|
||||
This is the section where we install and set up the new Redis Cache instances.
|
||||
|
||||
|
|
@ -832,10 +811,14 @@ a node and change its status from primary to replica (and vice versa).
|
|||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
# Specify server role as 'redis_master_role' and enable Consul agent
|
||||
roles(['redis_master_role', 'consul_role']
|
||||
# Specify server role as 'redis_sentinel_role' 'redis_master_role'
|
||||
roles ['redis_sentinel_role', 'redis_master_role', 'consul_role']
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach to.
|
||||
# Set IP bind address and Quorum number for Redis Sentinel service
|
||||
sentinel['bind'] = '0.0.0.0'
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach.
|
||||
# You can also set bind to '0.0.0.0' which listen in all interfaces.
|
||||
# If you really need to bind to an external accessible IP, make
|
||||
# sure you add extra firewall rules to prevent unauthorized access.
|
||||
|
|
@ -845,8 +828,18 @@ a node and change its status from primary to replica (and vice versa).
|
|||
# machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
# Port of primary Redis server for Sentinel, uncomment to change to non default.
|
||||
# Defaults to `6379`.
|
||||
# redis['master_port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis (use the same password in all nodes).
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
|
||||
# Must be the same in every Redis node.
|
||||
redis['master_name'] = 'gitlab-redis-cache'
|
||||
|
||||
# The IP of this primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.51'
|
||||
|
||||
# Set the Redis Cache instance as an LRU
|
||||
# 90% of available RAM in MB
|
||||
|
|
@ -880,10 +873,6 @@ a node and change its status from primary to replica (and vice versa).
|
|||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
You can specify multiple roles, like sentinel and Redis, as:
|
||||
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
|
||||
[roles](https://docs.gitlab.com/omnibus/roles/).
|
||||
|
||||
#### Configure the replica Redis Cache nodes
|
||||
|
||||
1. SSH in to the **replica** Redis server.
|
||||
|
|
@ -891,14 +880,18 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
|
||||
on the page, and to select the correct Omnibus GitLab package, with the same version
|
||||
and type (Community or Enterprise editions) as your current install.
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`:
|
||||
|
||||
```ruby
|
||||
# Specify server role as 'redis_replica_role' and enable Consul agent
|
||||
roles(['redis_replica_role', 'consul_role']
|
||||
roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role']
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach to.
|
||||
# You can also set bind to '0.0.0.0' which listen in all interfaces.
|
||||
# Set IP bind address and Quorum number for Redis Sentinel service
|
||||
sentinel['bind'] = `0.0.0.0`
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach.
|
||||
# Set bind to '0.0.0.0' to listen on all interfaces.
|
||||
# If you really need to bind to an external accessible IP, make
|
||||
# sure you add extra firewall rules to prevent unauthorized access.
|
||||
redis['bind'] = '10.6.0.52'
|
||||
|
|
@ -907,16 +900,19 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
# machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
# The same password for Redis authentication you set up for the primary node.
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis and replicas (use the same password in all nodes).
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
|
||||
# Must be the same in every Redis node
|
||||
redis['master_name'] = 'gitlab-redis-cache'
|
||||
|
||||
# The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.51'
|
||||
|
||||
# Port of primary Redis server, uncomment to change to non default. Defaults
|
||||
# to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
# Set the Redis Cache instance as an LRU
|
||||
# 90% of available RAM in MB
|
||||
redis['maxmemory'] = '13500mb'
|
||||
|
|
@ -952,17 +948,7 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
1. Go through the steps again for all the other replica nodes, and
|
||||
make sure to set up the IPs correctly.
|
||||
|
||||
You can specify multiple roles, like sentinel and Redis, as:
|
||||
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
|
||||
[roles](https://docs.gitlab.com/omnibus/roles/).
|
||||
|
||||
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
|
||||
a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a
|
||||
`gitlab-ctl reconfigure`, they will get their configuration restored by
|
||||
the same Sentinels.
|
||||
|
||||
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
|
||||
are supported and can be added if needed.
|
||||
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed.
|
||||
|
||||
<div align="right">
|
||||
<a type="button" class="btn btn-default" href="#setup-components">
|
||||
|
|
@ -970,134 +956,15 @@ are supported and can be added if needed.
|
|||
</a>
|
||||
</div>
|
||||
|
||||
#### Configure the Sentinel Cache nodes
|
||||
### Configure the Redis Persistent cluster
|
||||
|
||||
Now that the Redis servers are all set up, let's configure the Sentinel
|
||||
servers. The following IPs will be used as an example:
|
||||
|
||||
- `10.6.0.71`: Sentinel - Cache 1
|
||||
- `10.6.0.72`: Sentinel - Cache 2
|
||||
- `10.6.0.73`: Sentinel - Cache 3
|
||||
|
||||
NOTE:
|
||||
If you're using an external Redis Sentinel instance, be sure to exclude the
|
||||
`requirepass` parameter from the Sentinel configuration. This parameter causes
|
||||
clients to report `NOAUTH Authentication required.`.
|
||||
[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279).
|
||||
|
||||
To configure the Sentinel Cache server:
|
||||
|
||||
1. SSH in to the server that will host Consul/Sentinel.
|
||||
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
|
||||
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
|
||||
on the page, and to select the correct Omnibus GitLab package, with the same version
|
||||
and type (Community or Enterprise editions) as your current install.
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
roles(['redis_sentinel_role', 'consul_role'])
|
||||
|
||||
## Must be the same in every sentinel node
|
||||
redis['master_name'] = 'gitlab-redis-cache'
|
||||
|
||||
## The same password for Redis authentication you set up for the primary node.
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
|
||||
## The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.51'
|
||||
|
||||
## Define a port so Redis can listen for TCP requests which will allow other
|
||||
## machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
## Port of primary Redis server, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
## Configure Sentinel's IP
|
||||
sentinel['bind'] = '10.6.0.71'
|
||||
|
||||
## Port that Sentinel listens on, uncomment to change to non default. Defaults
|
||||
## to `26379`.
|
||||
#sentinel['port'] = 26379
|
||||
|
||||
## Quorum must reflect the amount of voting sentinels it take to start a failover.
|
||||
## Value must NOT be greater then the amount of sentinels.
|
||||
##
|
||||
## The quorum can be used to tune Sentinel in two ways:
|
||||
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
|
||||
## we deploy, we are basically making Sentinel more sensible to primary failures,
|
||||
## triggering a failover as soon as even just a minority of Sentinels is no longer
|
||||
## able to talk with the primary.
|
||||
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
|
||||
## making Sentinel able to failover only when there are a very large number (larger
|
||||
## than majority) of well connected Sentinels which agree about the primary being down.s
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
## Consider unresponsive server down after x amount of ms.
|
||||
#sentinel['down_after_milliseconds'] = 10000
|
||||
|
||||
## Specifies the failover timeout in milliseconds. It is used in many ways:
|
||||
##
|
||||
## - The time needed to re-start a failover after a previous failover was
|
||||
## already tried against the same primary by a given Sentinel, is two
|
||||
## times the failover timeout.
|
||||
##
|
||||
## - The time needed for a replica replicating to a wrong primary according
|
||||
## to a Sentinel current configuration, to be forced to replicate
|
||||
## with the right primary, is exactly the failover timeout (counting since
|
||||
## the moment a Sentinel detected the misconfiguration).
|
||||
##
|
||||
## - The time needed to cancel a failover that is already in progress but
|
||||
## did not produced any configuration change (REPLICAOF NO ONE yet not
|
||||
## acknowledged by the promoted replica).
|
||||
##
|
||||
## - The maximum time a failover in progress waits for all the replica to be
|
||||
## reconfigured as replicas of the new primary. However even after this time
|
||||
## the replicas will be reconfigured by the Sentinels anyway, but not with
|
||||
## the exact parallel-syncs progression as specified.
|
||||
#sentinel['failover_timeout'] = 60000
|
||||
|
||||
## Enable service discovery for Prometheus
|
||||
consul['monitoring_service_discovery'] = true
|
||||
|
||||
## The IPs of the Consul server nodes
|
||||
## You can also use FQDNs and intermix them with IPs
|
||||
consul['configuration'] = {
|
||||
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
|
||||
}
|
||||
|
||||
# Set the network addresses that the exporters will listen on
|
||||
node_exporter['listen_address'] = '0.0.0.0:9100'
|
||||
redis_exporter['listen_address'] = '0.0.0.0:9121'
|
||||
|
||||
# Prevent database migrations from running on upgrade automatically
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
```
|
||||
|
||||
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
|
||||
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
|
||||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
1. Go through the steps again for all the other Consul/Sentinel nodes, and
|
||||
make sure you set up the correct IPs.
|
||||
|
||||
<div align="right">
|
||||
<a type="button" class="btn btn-default" href="#setup-components">
|
||||
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
### Configure the Redis and Sentinel Queues cluster
|
||||
|
||||
This is the section where we install and set up the new Redis Queues instances.
|
||||
This is the section where we install and set up the new Redis Persistent instances.
|
||||
|
||||
Both the primary and replica Redis nodes need the same password defined in
|
||||
`redis['password']`. At any time during a failover, the Sentinels can reconfigure
|
||||
a node and change its status from primary to replica (and vice versa).
|
||||
|
||||
#### Configure the primary Redis Queues node
|
||||
#### Configure the primary Redis Persistent node
|
||||
|
||||
1. SSH in to the **Primary** Redis server.
|
||||
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
|
||||
|
|
@ -1107,8 +974,12 @@ a node and change its status from primary to replica (and vice versa).
|
|||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
# Specify server role as 'redis_master_role' and enable Consul agent
|
||||
roles(['redis_master_role', 'consul_role'])
|
||||
# Specify server roles as 'redis_sentinel_role' and 'redis_master_role'
|
||||
roles ['redis_sentinel_role', 'redis_master_role', 'consul_role']
|
||||
|
||||
# Set IP bind address and Quorum number for Redis Sentinel service
|
||||
sentinel['bind'] = '0.0.0.0'
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach to.
|
||||
# You can also set bind to '0.0.0.0' which listen in all interfaces.
|
||||
|
|
@ -1120,8 +991,19 @@ a node and change its status from primary to replica (and vice versa).
|
|||
# machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis (use the same password in all nodes).
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis and replicas (use the same password in all nodes).
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
|
||||
## Must be the same in every Redis node
|
||||
redis['master_name'] = 'gitlab-redis-persistent'
|
||||
|
||||
## The IP of this primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.61'
|
||||
|
||||
## Enable service discovery for Prometheus
|
||||
consul['monitoring_service_discovery'] = true
|
||||
|
|
@ -1145,13 +1027,9 @@ a node and change its status from primary to replica (and vice versa).
|
|||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
You can specify multiple roles, like sentinel and Redis, as:
|
||||
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
|
||||
[roles](https://docs.gitlab.com/omnibus/roles/).
|
||||
#### Configure the replica Redis Persistent nodes
|
||||
|
||||
#### Configure the replica Redis Queues nodes
|
||||
|
||||
1. SSH in to the **replica** Redis Queue server.
|
||||
1. SSH in to the **replica** Redis Persistent server.
|
||||
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
|
||||
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
|
||||
on the page, and to select the correct Omnibus GitLab package, with the same version
|
||||
|
|
@ -1160,7 +1038,11 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
|
||||
```ruby
|
||||
# Specify server role as 'redis_replica_role' and enable Consul agent
|
||||
roles(['redis_replica_role', 'consul_role'])
|
||||
roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role']
|
||||
|
||||
# Set IP bind address and Quorum number for Redis Sentinel service
|
||||
sentinel['bind'] = '0.0.0.0'
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach to.
|
||||
# You can also set bind to '0.0.0.0' which listen in all interfaces.
|
||||
|
|
@ -1172,16 +1054,19 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
# machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
# The same password for Redis authentication you set up for the primary node.
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
|
||||
## Must be the same in every Redis node
|
||||
redis['master_name'] = 'gitlab-redis-persistent'
|
||||
|
||||
# The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.61'
|
||||
|
||||
# Port of primary Redis server, uncomment to change to non default. Defaults
|
||||
# to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
## Enable service discovery for Prometheus
|
||||
consul['monitoring_service_discovery'] = true
|
||||
|
||||
|
|
@ -1211,15 +1096,6 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
1. Go through the steps again for all the other replica nodes, and
|
||||
make sure to set up the IPs correctly.
|
||||
|
||||
You can specify multiple roles, like sentinel and Redis, as:
|
||||
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
|
||||
[roles](https://docs.gitlab.com/omnibus/roles/).
|
||||
|
||||
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
|
||||
a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a
|
||||
`gitlab-ctl reconfigure`, they will get their configuration restored by
|
||||
the same Sentinels.
|
||||
|
||||
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
|
||||
are supported and can be added if needed.
|
||||
|
||||
|
|
@ -1229,136 +1105,10 @@ are supported and can be added if needed.
|
|||
</a>
|
||||
</div>
|
||||
|
||||
#### Configure the Sentinel Queues nodes
|
||||
|
||||
Now that the Redis servers are all set up, let's configure the Sentinel
|
||||
servers. The following IPs will be used as an example:
|
||||
|
||||
- `10.6.0.81`: Sentinel - Queues 1
|
||||
- `10.6.0.82`: Sentinel - Queues 2
|
||||
- `10.6.0.83`: Sentinel - Queues 3
|
||||
|
||||
NOTE:
|
||||
If you're using an external Redis Sentinel instance, be sure to exclude the
|
||||
`requirepass` parameter from the Sentinel configuration. This parameter causes
|
||||
clients to report `NOAUTH Authentication required.`.
|
||||
[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279).
|
||||
|
||||
To configure the Sentinel Queues server:
|
||||
|
||||
1. SSH in to the server that will host Sentinel.
|
||||
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
|
||||
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
|
||||
on the page, and to select the correct Omnibus GitLab package, with the same version
|
||||
and type (Community or Enterprise editions) as your current install.
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
roles(['redis_sentinel_role', 'consul_role'])
|
||||
|
||||
## Must be the same in every sentinel node
|
||||
redis['master_name'] = 'gitlab-redis-persistent'
|
||||
|
||||
## The same password for Redis authentication you set up for the primary node.
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
|
||||
## The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.61'
|
||||
|
||||
## Define a port so Redis can listen for TCP requests which will allow other
|
||||
## machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
## Port of primary Redis server, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
## Configure Sentinel's IP
|
||||
sentinel['bind'] = '10.6.0.81'
|
||||
|
||||
## Port that Sentinel listens on, uncomment to change to non default. Defaults
|
||||
## to `26379`.
|
||||
#sentinel['port'] = 26379
|
||||
|
||||
## Quorum must reflect the amount of voting sentinels it take to start a failover.
|
||||
## Value must NOT be greater then the amount of sentinels.
|
||||
##
|
||||
## The quorum can be used to tune Sentinel in two ways:
|
||||
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
|
||||
## we deploy, we are basically making Sentinel more sensible to primary failures,
|
||||
## triggering a failover as soon as even just a minority of Sentinels is no longer
|
||||
## able to talk with the primary.
|
||||
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
|
||||
## making Sentinel able to failover only when there are a very large number (larger
|
||||
## than majority) of well connected Sentinels which agree about the primary being down.s
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
## Consider unresponsive server down after x amount of ms.
|
||||
#sentinel['down_after_milliseconds'] = 10000
|
||||
|
||||
## Specifies the failover timeout in milliseconds. It is used in many ways:
|
||||
##
|
||||
## - The time needed to re-start a failover after a previous failover was
|
||||
## already tried against the same primary by a given Sentinel, is two
|
||||
## times the failover timeout.
|
||||
##
|
||||
## - The time needed for a replica replicating to a wrong primary according
|
||||
## to a Sentinel current configuration, to be forced to replicate
|
||||
## with the right primary, is exactly the failover timeout (counting since
|
||||
## the moment a Sentinel detected the misconfiguration).
|
||||
##
|
||||
## - The time needed to cancel a failover that is already in progress but
|
||||
## did not produced any configuration change (REPLICAOF NO ONE yet not
|
||||
## acknowledged by the promoted replica).
|
||||
##
|
||||
## - The maximum time a failover in progress waits for all the replica to be
|
||||
## reconfigured as replicas of the new primary. However even after this time
|
||||
## the replicas will be reconfigured by the Sentinels anyway, but not with
|
||||
## the exact parallel-syncs progression as specified.
|
||||
#sentinel['failover_timeout'] = 60000
|
||||
|
||||
## Enable service discovery for Prometheus
|
||||
consul['monitoring_service_discovery'] = true
|
||||
|
||||
## The IPs of the Consul server nodes
|
||||
## You can also use FQDNs and intermix them with IPs
|
||||
consul['configuration'] = {
|
||||
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
|
||||
}
|
||||
|
||||
# Set the network addresses that the exporters will listen on
|
||||
node_exporter['listen_address'] = '0.0.0.0:9100'
|
||||
redis_exporter['listen_address'] = '0.0.0.0:9121'
|
||||
|
||||
# Prevent database migrations from running on upgrade automatically
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
```
|
||||
|
||||
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
|
||||
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
|
||||
|
||||
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
|
||||
|
||||
```shell
|
||||
sudo touch /etc/gitlab/skip-auto-reconfigure
|
||||
```
|
||||
|
||||
Only the primary GitLab application server should handle migrations.
|
||||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
1. Go through the steps again for all the other Sentinel nodes, and
|
||||
make sure you set up the correct IPs.
|
||||
|
||||
<div align="right">
|
||||
<a type="button" class="btn btn-default" href="#setup-components">
|
||||
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
## Configure Gitaly Cluster
|
||||
|
||||
[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories.
|
||||
[Gitaly Cluster](../gitaly/praefect.md) is a GitLab-provided and recommended
|
||||
fault tolerant solution for storing Git repositories.
|
||||
In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down.
|
||||
|
||||
The recommended cluster setup includes the following components:
|
||||
|
|
@ -1875,30 +1625,30 @@ To configure the Sidekiq nodes, on each one:
|
|||
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
|
||||
|
||||
gitlab_rails['redis_cache_sentinels'] = [
|
||||
{host: '10.6.0.71', port: 26379},
|
||||
{host: '10.6.0.72', port: 26379},
|
||||
{host: '10.6.0.73', port: 26379},
|
||||
{host: '10.6.0.51', port: 26379},
|
||||
{host: '10.6.0.52', port: 26379},
|
||||
{host: '10.6.0.53', port: 26379},
|
||||
]
|
||||
|
||||
## Second cluster that will host the queues, shared state, and actioncable
|
||||
## Second cluster that will host the persistent queues, shared state, and actioncable
|
||||
gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
|
||||
gitlab_rails['redis_queues_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
gitlab_rails['redis_shared_state_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
gitlab_rails['redis_actioncable_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
|
||||
# Gitaly Cluster
|
||||
|
|
@ -2052,9 +1802,9 @@ On each node perform the following:
|
|||
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
|
||||
|
||||
gitlab_rails['redis_cache_sentinels'] = [
|
||||
{host: '10.6.0.71', port: 26379},
|
||||
{host: '10.6.0.72', port: 26379},
|
||||
{host: '10.6.0.73', port: 26379},
|
||||
{host: '10.6.0.51', port: 26379},
|
||||
{host: '10.6.0.52', port: 26379},
|
||||
{host: '10.6.0.53', port: 26379},
|
||||
]
|
||||
|
||||
## Second cluster that will host the queues, shared state, and actionable
|
||||
|
|
@ -2063,19 +1813,19 @@ On each node perform the following:
|
|||
gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
|
||||
gitlab_rails['redis_queues_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
gitlab_rails['redis_shared_state_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
gitlab_rails['redis_actioncable_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
|
||||
# Set the network addresses that the exporters used for monitoring will listen on
|
||||
|
|
@ -2147,7 +1897,7 @@ On each node perform the following:
|
|||
|
||||
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
1. If you're [using NFS](#configure-nfs-optional):
|
||||
1. If you're [using NFS](#configure-nfs):
|
||||
1. If necessary, install the NFS client utility packages using the following
|
||||
commands:
|
||||
|
||||
|
|
@ -2289,9 +2039,9 @@ To configure the Monitoring node:
|
|||
## Configure the object storage
|
||||
|
||||
GitLab supports using an object storage service for holding numerous types of data.
|
||||
It's recommended over [NFS](#configure-nfs-optional) and in general it's better
|
||||
in larger setups as object storage is typically much more performant, reliable,
|
||||
and scalable.
|
||||
Object storage is also recommended over [NFS](#configure-nfs) and in general
|
||||
it's better in larger setups as object storage is typically much more performant,
|
||||
reliable, and scalable.
|
||||
|
||||
GitLab has been tested on a number of object storage providers:
|
||||
|
||||
|
|
@ -2359,7 +2109,7 @@ cluster alongside your instance, read how to
|
|||
</a>
|
||||
</div>
|
||||
|
||||
## Configure NFS (optional)
|
||||
## Configure NFS
|
||||
|
||||
[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly)
|
||||
are recommended over NFS wherever possible for improved performance. If you intend
|
||||
|
|
@ -2427,10 +2177,8 @@ services where applicable):
|
|||
| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` |
|
||||
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
|
||||
| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` |
|
||||
| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
|
||||
| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
|
||||
| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` |
|
||||
| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` |
|
||||
| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
|
||||
| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
|
||||
| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` |
|
||||
| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` |
|
||||
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
|
||||
|
|
@ -2486,11 +2234,6 @@ card "Database" as database {
|
|||
card "redis" as redis {
|
||||
collections "**Redis Persistent** x3" as redis_persistent #FF6347
|
||||
collections "**Redis Cache** x3" as redis_cache #FF6347
|
||||
collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347
|
||||
collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347
|
||||
|
||||
redis_persistent <.[#FF6347]- redis_persistent_sentinel
|
||||
redis_cache <.[#FF6347]- redis_cache_sentinel
|
||||
}
|
||||
|
||||
cloud "**Object Storage**" as object_storage #white
|
||||
|
|
|
|||
|
|
@ -22,10 +22,8 @@ full list of reference architectures, see
|
|||
| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` |
|
||||
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
|
||||
| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` |
|
||||
| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
|
||||
| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
|
||||
| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` |
|
||||
| Redis Sentinel - Queues / Shared State<sup>2</sup>| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` |
|
||||
| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
|
||||
| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` |
|
||||
| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` |
|
||||
| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
|
||||
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` |
|
||||
|
|
@ -33,7 +31,7 @@ full list of reference architectures, see
|
|||
| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` |
|
||||
| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
|
||||
| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a |
|
||||
| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
|
||||
| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` |
|
||||
|
||||
<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix -->
|
||||
<!-- markdownlint-disable MD029 -->
|
||||
|
|
@ -82,11 +80,6 @@ card "Database" as database {
|
|||
card "redis" as redis {
|
||||
collections "**Redis Persistent** x3" as redis_persistent #FF6347
|
||||
collections "**Redis Cache** x3" as redis_cache #FF6347
|
||||
collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347
|
||||
collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347
|
||||
|
||||
redis_persistent <.[#FF6347]- redis_persistent_sentinel
|
||||
redis_cache <.[#FF6347]- redis_cache_sentinel
|
||||
}
|
||||
|
||||
cloud "**Object Storage**" as object_storage #white
|
||||
|
|
@ -137,8 +130,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based
|
|||
|
||||
Due to better performance and availability, for data objects (such as LFS,
|
||||
uploads, or artifacts), using an [object storage service](#configure-the-object-storage)
|
||||
is recommended instead of using NFS. Using an object storage service also
|
||||
doesn't require you to provision and maintain a node.
|
||||
is recommended.
|
||||
|
||||
It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and
|
||||
that to achieve full High Availability a third-party PostgreSQL database solution will be required.
|
||||
|
|
@ -169,10 +161,8 @@ To set up GitLab and its components to accommodate up to 50,000 users:
|
|||
used for shared data objects.
|
||||
1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster,
|
||||
more advanced code search across your entire GitLab instance.
|
||||
1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended)
|
||||
to have shared disk storage service as an alternative to Gitaly or object
|
||||
storage. You can skip this step if you're not using GitLab Pages (which
|
||||
requires NFS).
|
||||
1. [Configure NFS](#configure-nfs)
|
||||
to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage).
|
||||
|
||||
The servers start on the same 10.6.0.0/24 private network range, and can
|
||||
connect to each other freely on these addresses.
|
||||
|
|
@ -193,15 +183,9 @@ The following list includes descriptions of each server and its assigned IP:
|
|||
- `10.6.0.51`: Redis - Cache Primary
|
||||
- `10.6.0.52`: Redis - Cache Replica 1
|
||||
- `10.6.0.53`: Redis - Cache Replica 2
|
||||
- `10.6.0.71`: Sentinel - Cache 1
|
||||
- `10.6.0.72`: Sentinel - Cache 2
|
||||
- `10.6.0.73`: Sentinel - Cache 3
|
||||
- `10.6.0.61`: Redis - Queues Primary
|
||||
- `10.6.0.62`: Redis - Queues Replica 1
|
||||
- `10.6.0.63`: Redis - Queues Replica 2
|
||||
- `10.6.0.81`: Sentinel - Queues 1
|
||||
- `10.6.0.82`: Sentinel - Queues 2
|
||||
- `10.6.0.83`: Sentinel - Queues 3
|
||||
- `10.6.0.61`: Redis - Persistent Primary
|
||||
- `10.6.0.62`: Redis - Persistent Replica 1
|
||||
- `10.6.0.63`: Redis - Persistent Replica 2
|
||||
- `10.6.0.91`: Gitaly 1
|
||||
- `10.6.0.92`: Gitaly 2
|
||||
- `10.6.0.93`: Gitaly 3
|
||||
|
|
@ -802,15 +786,9 @@ to be used with GitLab. The following IPs will be used as an example:
|
|||
- `10.6.0.51`: Redis - Cache Primary
|
||||
- `10.6.0.52`: Redis - Cache Replica 1
|
||||
- `10.6.0.53`: Redis - Cache Replica 2
|
||||
- `10.6.0.71`: Sentinel - Cache 1
|
||||
- `10.6.0.72`: Sentinel - Cache 2
|
||||
- `10.6.0.73`: Sentinel - Cache 3
|
||||
- `10.6.0.61`: Redis - Queues Primary
|
||||
- `10.6.0.62`: Redis - Queues Replica 1
|
||||
- `10.6.0.63`: Redis - Queues Replica 2
|
||||
- `10.6.0.81`: Sentinel - Queues 1
|
||||
- `10.6.0.82`: Sentinel - Queues 2
|
||||
- `10.6.0.83`: Sentinel - Queues 3
|
||||
- `10.6.0.61`: Redis - Persistent Primary
|
||||
- `10.6.0.62`: Redis - Persistent Replica 1
|
||||
- `10.6.0.63`: Redis - Persistent Replica 2
|
||||
|
||||
### Providing your own Redis instance
|
||||
|
||||
|
|
@ -822,7 +800,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p
|
|||
Note the Redis node's IP address or hostname, port, and password (if required).
|
||||
These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails).
|
||||
|
||||
### Configure the Redis and Sentinel Cache cluster
|
||||
### Configure the Redis Cache cluster
|
||||
|
||||
This is the section where we install and set up the new Redis Cache instances.
|
||||
|
||||
|
|
@ -840,8 +818,12 @@ a node and change its status from primary to replica (and vice versa).
|
|||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
# Specify server role as 'redis_master_role' and enable Consul agent
|
||||
roles(['redis_master_role', 'consul_role'])
|
||||
# Specify server role as 'redis_master_role' with Sentinel and enable Consul agent
|
||||
roles(['redis_sentinel_role', 'redis_master_role', 'consul_role'])
|
||||
|
||||
# Set IP bind address and Quorum number for Redis Sentinel service
|
||||
sentinel['bind'] = '0.0.0.0'
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach to.
|
||||
# You can also set bind to '0.0.0.0' which listen in all interfaces.
|
||||
|
|
@ -853,8 +835,19 @@ a node and change its status from primary to replica (and vice versa).
|
|||
# machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis (use the same password in all nodes).
|
||||
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis and replicas (use the same password in all nodes).
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
|
||||
## Must be the same in every Redis node
|
||||
redis['master_name'] = 'gitlab-redis-cache'
|
||||
|
||||
## The IP of this primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.51'
|
||||
|
||||
# Set the Redis Cache instance as an LRU
|
||||
# 90% of available RAM in MB
|
||||
|
|
@ -888,10 +881,6 @@ a node and change its status from primary to replica (and vice versa).
|
|||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
You can specify multiple roles, like sentinel and Redis, as:
|
||||
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
|
||||
[roles](https://docs.gitlab.com/omnibus/roles/).
|
||||
|
||||
#### Configure the replica Redis Cache nodes
|
||||
|
||||
1. SSH in to the **replica** Redis server.
|
||||
|
|
@ -899,11 +888,15 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
|
||||
on the page, and to select the correct Omnibus GitLab package, with the same version
|
||||
and type (Community or Enterprise editions) as your current install.
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the priimary node in the previous section by replacing `redis_master_node` with `redis_replica_node`:
|
||||
|
||||
```ruby
|
||||
# Specify server role as 'redis_replica_role' and enable Consul agent
|
||||
roles(['redis_replica_role', 'consul_role'])
|
||||
# Specify server role as 'redis_replica_role' with Sentinel and enable Consul agent
|
||||
roles(['roles_sentinel_role', 'redis_replica_role', 'consul_role'])
|
||||
|
||||
# Set IP bind address and Quorum number for Redis Sentinel service
|
||||
sentinel['bind'] = '0.0.0.0'
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach to.
|
||||
# You can also set bind to '0.0.0.0' which listen in all interfaces.
|
||||
|
|
@ -915,16 +908,20 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
# machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
# The same password for Redis authentication you set up for the primary node.
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
|
||||
# The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.51'
|
||||
|
||||
# Port of primary Redis server, uncomment to change to non default. Defaults
|
||||
# to `6379`.
|
||||
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis and replicas (use the same password in all nodes).
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
|
||||
## Must be the same in every Redis node
|
||||
redis['master_name'] = 'gitlab-redis-cache'
|
||||
|
||||
## The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.51'
|
||||
|
||||
# Set the Redis Cache instance as an LRU
|
||||
# 90% of available RAM in MB
|
||||
redis['maxmemory'] = '13500mb'
|
||||
|
|
@ -952,25 +949,18 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
gitlab_rails['auto_migrate'] = false
|
||||
```
|
||||
|
||||
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
|
||||
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
|
||||
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus
|
||||
node you configured and add or replace the file of the same name on this
|
||||
server. If this is the first Omnibus node you are configuring then you
|
||||
can skip this step.
|
||||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes
|
||||
to take effect.
|
||||
|
||||
1. Go through the steps again for all the other replica nodes, and
|
||||
1. Go through the steps again for all the other replica nodes, and
|
||||
make sure to set up the IPs correctly.
|
||||
|
||||
You can specify multiple roles, like sentinel and Redis, as:
|
||||
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
|
||||
[roles](https://docs.gitlab.com/omnibus/roles/).
|
||||
|
||||
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
|
||||
a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a
|
||||
`gitlab-ctl reconfigure`, they will get their configuration restored by
|
||||
the same Sentinels.
|
||||
|
||||
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
|
||||
are supported and can be added if needed.
|
||||
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed.
|
||||
|
||||
<div align="right">
|
||||
<a type="button" class="btn btn-default" href="#setup-components">
|
||||
|
|
@ -978,126 +968,7 @@ are supported and can be added if needed.
|
|||
</a>
|
||||
</div>
|
||||
|
||||
#### Configure the Sentinel Cache nodes
|
||||
|
||||
Now that the Redis servers are all set up, let's configure the Sentinel
|
||||
servers. The following IPs will be used as an example:
|
||||
|
||||
- `10.6.0.71`: Sentinel - Cache 1
|
||||
- `10.6.0.72`: Sentinel - Cache 2
|
||||
- `10.6.0.73`: Sentinel - Cache 3
|
||||
|
||||
NOTE:
|
||||
If you're using an external Redis Sentinel instance, be sure to exclude the
|
||||
`requirepass` parameter from the Sentinel configuration. This parameter causes
|
||||
clients to report `NOAUTH Authentication required.`.
|
||||
[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279).
|
||||
|
||||
To configure the Sentinel Cache server:
|
||||
|
||||
1. SSH in to the server that will host Consul/Sentinel.
|
||||
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
|
||||
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
|
||||
on the page, and to select the correct Omnibus GitLab package, with the same version
|
||||
and type (Community or Enterprise editions) as your current install.
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
roles(['redis_sentinel_role', 'consul_role'])
|
||||
|
||||
## Must be the same in every sentinel node
|
||||
redis['master_name'] = 'gitlab-redis-cache'
|
||||
|
||||
## The same password for Redis authentication you set up for the primary node.
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
|
||||
## The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.51'
|
||||
|
||||
## Define a port so Redis can listen for TCP requests which will allow other
|
||||
## machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
## Port of primary Redis server, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
## Configure Sentinel's IP
|
||||
sentinel['bind'] = '10.6.0.71'
|
||||
|
||||
## Port that Sentinel listens on, uncomment to change to non default. Defaults
|
||||
## to `26379`.
|
||||
#sentinel['port'] = 26379
|
||||
|
||||
## Quorum must reflect the amount of voting sentinels it take to start a failover.
|
||||
## Value must NOT be greater then the amount of sentinels.
|
||||
##
|
||||
## The quorum can be used to tune Sentinel in two ways:
|
||||
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
|
||||
## we deploy, we are basically making Sentinel more sensible to primary failures,
|
||||
## triggering a failover as soon as even just a minority of Sentinels is no longer
|
||||
## able to talk with the primary.
|
||||
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
|
||||
## making Sentinel able to failover only when there are a very large number (larger
|
||||
## than majority) of well connected Sentinels which agree about the primary being down.s
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
## Consider unresponsive server down after x amount of ms.
|
||||
#sentinel['down_after_milliseconds'] = 10000
|
||||
|
||||
## Specifies the failover timeout in milliseconds. It is used in many ways:
|
||||
##
|
||||
## - The time needed to re-start a failover after a previous failover was
|
||||
## already tried against the same primary by a given Sentinel, is two
|
||||
## times the failover timeout.
|
||||
##
|
||||
## - The time needed for a replica replicating to a wrong primary according
|
||||
## to a Sentinel current configuration, to be forced to replicate
|
||||
## with the right primary, is exactly the failover timeout (counting since
|
||||
## the moment a Sentinel detected the misconfiguration).
|
||||
##
|
||||
## - The time needed to cancel a failover that is already in progress but
|
||||
## did not produced any configuration change (REPLICAOF NO ONE yet not
|
||||
## acknowledged by the promoted replica).
|
||||
##
|
||||
## - The maximum time a failover in progress waits for all the replica to be
|
||||
## reconfigured as replicas of the new primary. However even after this time
|
||||
## the replicas will be reconfigured by the Sentinels anyway, but not with
|
||||
## the exact parallel-syncs progression as specified.
|
||||
#sentinel['failover_timeout'] = 60000
|
||||
|
||||
## Enable service discovery for Prometheus
|
||||
consul['monitoring_service_discovery'] = true
|
||||
|
||||
## The IPs of the Consul server nodes
|
||||
## You can also use FQDNs and intermix them with IPs
|
||||
consul['configuration'] = {
|
||||
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
|
||||
}
|
||||
|
||||
# Set the network addresses that the exporters will listen on
|
||||
node_exporter['listen_address'] = '0.0.0.0:9100'
|
||||
redis_exporter['listen_address'] = '0.0.0.0:9121'
|
||||
|
||||
# Prevent database migrations from running on upgrade automatically
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
```
|
||||
|
||||
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
|
||||
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
|
||||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
1. Go through the steps again for all the other Consul/Sentinel nodes, and
|
||||
make sure you set up the correct IPs.
|
||||
|
||||
<div align="right">
|
||||
<a type="button" class="btn btn-default" href="#setup-components">
|
||||
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
### Configure the Redis and Sentinel Queues cluster
|
||||
### Configure the Redis Persistent cluster
|
||||
|
||||
This is the section where we install and set up the new Redis Queues instances.
|
||||
|
||||
|
|
@ -1105,7 +976,7 @@ Both the primary and replica Redis nodes need the same password defined in
|
|||
`redis['password']`. At any time during a failover, the Sentinels can reconfigure
|
||||
a node and change its status from primary to replica (and vice versa).
|
||||
|
||||
#### Configure the primary Redis Queues node
|
||||
#### Configure the primary Redis Persistent node
|
||||
|
||||
1. SSH in to the **Primary** Redis server.
|
||||
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
|
||||
|
|
@ -1115,8 +986,12 @@ a node and change its status from primary to replica (and vice versa).
|
|||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
# Specify server role as 'redis_master_role' and enable Consul agent
|
||||
roles(['redis_master_role', 'consul_role'])
|
||||
# Specify server roles as 'redis_master_role' with Sentinel and enable the Consul agent
|
||||
roles ['redis_sentinel_role', 'redis_master_role', 'consul_role']
|
||||
|
||||
# Set IP bind address and Quorum number for Redis Sentinel service
|
||||
sentinel['bind'] = '0.0.0.0'
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach to.
|
||||
# You can also set bind to '0.0.0.0' which listen in all interfaces.
|
||||
|
|
@ -1128,8 +1003,19 @@ a node and change its status from primary to replica (and vice versa).
|
|||
# machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis (use the same password in all nodes).
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
# Set up password authentication for Redis and replicas (use the same password in all nodes).
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER'
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
|
||||
## Must be the same in every Redis node
|
||||
redis['master_name'] = 'gitlab-redis-persistent'
|
||||
|
||||
## The IP of this primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.61'
|
||||
|
||||
## Enable service discovery for Prometheus
|
||||
consul['monitoring_service_discovery'] = true
|
||||
|
|
@ -1153,13 +1039,9 @@ a node and change its status from primary to replica (and vice versa).
|
|||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
You can specify multiple roles, like sentinel and Redis, as:
|
||||
`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about
|
||||
[roles](https://docs.gitlab.com/omnibus/roles/).
|
||||
#### Configure the replica Redis Persistent nodes
|
||||
|
||||
#### Configure the replica Redis Queues nodes
|
||||
|
||||
1. SSH in to the **replica** Redis Queue server.
|
||||
1. SSH in to the **replica** Redis Persistent server.
|
||||
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
|
||||
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
|
||||
on the page, and to select the correct Omnibus GitLab package, with the same version
|
||||
|
|
@ -1167,8 +1049,12 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
# Specify server role as 'redis_replica_role' and enable Consul agent
|
||||
roles(['redis_replica_role', 'consul_role'])
|
||||
# Specify server roles as 'redis_replica_role' with Sentinel and enable Consul agent
|
||||
roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role']
|
||||
|
||||
# Set IP bind address and Quorum number for Redis Sentinel service
|
||||
sentinel['bind'] = '0.0.0.0'
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
# IP address pointing to a local IP that the other machines can reach to.
|
||||
# You can also set bind to '0.0.0.0' which listen in all interfaces.
|
||||
|
|
@ -1180,16 +1066,20 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
# machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
# The same password for Redis authentication you set up for the primary node.
|
||||
redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
|
||||
## Must be the same in every Redis node
|
||||
redis['master_name'] = 'gitlab-redis-persistent'
|
||||
|
||||
# The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.61'
|
||||
|
||||
# Port of primary Redis server, uncomment to change to non default. Defaults
|
||||
# to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
## Enable service discovery for Prometheus
|
||||
consul['monitoring_service_discovery'] = true
|
||||
|
||||
|
|
@ -1215,144 +1105,7 @@ You can specify multiple roles, like sentinel and Redis, as:
|
|||
1. Go through the steps again for all the other replica nodes, and
|
||||
make sure to set up the IPs correctly.
|
||||
|
||||
You can specify multiple roles, like sentinel and Redis, as:
|
||||
`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about
|
||||
[roles](https://docs.gitlab.com/omnibus/roles/).
|
||||
|
||||
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
|
||||
a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a
|
||||
`gitlab-ctl reconfigure`, they will get their configuration restored by
|
||||
the same Sentinels.
|
||||
|
||||
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
|
||||
are supported and can be added if needed.
|
||||
|
||||
<div align="right">
|
||||
<a type="button" class="btn btn-default" href="#setup-components">
|
||||
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
#### Configure the Sentinel Queues nodes
|
||||
|
||||
Now that the Redis servers are all set up, let's configure the Sentinel
|
||||
servers. The following IPs will be used as an example:
|
||||
|
||||
- `10.6.0.81`: Sentinel - Queues 1
|
||||
- `10.6.0.82`: Sentinel - Queues 2
|
||||
- `10.6.0.83`: Sentinel - Queues 3
|
||||
|
||||
NOTE:
|
||||
If you're using an external Redis Sentinel instance, be sure to exclude the
|
||||
`requirepass` parameter from the Sentinel configuration. This parameter causes
|
||||
clients to report `NOAUTH Authentication required.`.
|
||||
[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279).
|
||||
|
||||
To configure the Sentinel Queues server:
|
||||
|
||||
1. SSH in to the server that will host Sentinel.
|
||||
1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab
|
||||
package of your choice. Be sure to both follow _only_ installation steps 1 and 2
|
||||
on the page, and to select the correct Omnibus GitLab package, with the same version
|
||||
and type (Community or Enterprise editions) as your current install.
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
|
||||
|
||||
```ruby
|
||||
roles(['redis_sentinel_role', 'consul_role'])
|
||||
|
||||
## Must be the same in every sentinel node
|
||||
redis['master_name'] = 'gitlab-redis-persistent'
|
||||
|
||||
## The same password for Redis authentication you set up for the primary node.
|
||||
redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER'
|
||||
|
||||
## The IP of the primary Redis node.
|
||||
redis['master_ip'] = '10.6.0.61'
|
||||
|
||||
## Define a port so Redis can listen for TCP requests which will allow other
|
||||
## machines to connect to it.
|
||||
redis['port'] = 6379
|
||||
|
||||
## Port of primary Redis server, uncomment to change to non default. Defaults
|
||||
## to `6379`.
|
||||
#redis['master_port'] = 6379
|
||||
|
||||
## Configure Sentinel's IP
|
||||
sentinel['bind'] = '10.6.0.81'
|
||||
|
||||
## Port that Sentinel listens on, uncomment to change to non default. Defaults
|
||||
## to `26379`.
|
||||
#sentinel['port'] = 26379
|
||||
|
||||
## Quorum must reflect the amount of voting sentinels it take to start a failover.
|
||||
## Value must NOT be greater then the amount of sentinels.
|
||||
##
|
||||
## The quorum can be used to tune Sentinel in two ways:
|
||||
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
|
||||
## we deploy, we are basically making Sentinel more sensible to primary failures,
|
||||
## triggering a failover as soon as even just a minority of Sentinels is no longer
|
||||
## able to talk with the primary.
|
||||
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
|
||||
## making Sentinel able to failover only when there are a very large number (larger
|
||||
## than majority) of well connected Sentinels which agree about the primary being down.s
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
## Consider unresponsive server down after x amount of ms.
|
||||
#sentinel['down_after_milliseconds'] = 10000
|
||||
|
||||
## Specifies the failover timeout in milliseconds. It is used in many ways:
|
||||
##
|
||||
## - The time needed to re-start a failover after a previous failover was
|
||||
## already tried against the same primary by a given Sentinel, is two
|
||||
## times the failover timeout.
|
||||
##
|
||||
## - The time needed for a replica replicating to a wrong primary according
|
||||
## to a Sentinel current configuration, to be forced to replicate
|
||||
## with the right primary, is exactly the failover timeout (counting since
|
||||
## the moment a Sentinel detected the misconfiguration).
|
||||
##
|
||||
## - The time needed to cancel a failover that is already in progress but
|
||||
## did not produced any configuration change (REPLICAOF NO ONE yet not
|
||||
## acknowledged by the promoted replica).
|
||||
##
|
||||
## - The maximum time a failover in progress waits for all the replica to be
|
||||
## reconfigured as replicas of the new primary. However even after this time
|
||||
## the replicas will be reconfigured by the Sentinels anyway, but not with
|
||||
## the exact parallel-syncs progression as specified.
|
||||
#sentinel['failover_timeout'] = 60000
|
||||
|
||||
## Enable service discovery for Prometheus
|
||||
consul['monitoring_service_discovery'] = true
|
||||
|
||||
## The IPs of the Consul server nodes
|
||||
## You can also use FQDNs and intermix them with IPs
|
||||
consul['configuration'] = {
|
||||
retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13),
|
||||
}
|
||||
|
||||
# Set the network addresses that the exporters will listen on
|
||||
node_exporter['listen_address'] = '0.0.0.0:9100'
|
||||
redis_exporter['listen_address'] = '0.0.0.0:9121'
|
||||
|
||||
# Prevent database migrations from running on upgrade automatically
|
||||
gitlab_rails['auto_migrate'] = false
|
||||
```
|
||||
|
||||
1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run:
|
||||
|
||||
```shell
|
||||
sudo touch /etc/gitlab/skip-auto-reconfigure
|
||||
```
|
||||
|
||||
Only the primary GitLab application server should handle migrations.
|
||||
|
||||
1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace
|
||||
the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step.
|
||||
|
||||
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
1. Go through the steps again for all the other Sentinel nodes, and
|
||||
make sure you set up the correct IPs.
|
||||
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed.
|
||||
|
||||
<div align="right">
|
||||
<a type="button" class="btn btn-default" href="#setup-components">
|
||||
|
|
@ -1879,30 +1632,30 @@ To configure the Sidekiq nodes, on each one:
|
|||
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
|
||||
|
||||
gitlab_rails['redis_cache_sentinels'] = [
|
||||
{host: '10.6.0.71', port: 26379},
|
||||
{host: '10.6.0.72', port: 26379},
|
||||
{host: '10.6.0.73', port: 26379},
|
||||
{host: '10.6.0.51', port: 26379},
|
||||
{host: '10.6.0.52', port: 26379},
|
||||
{host: '10.6.0.53', port: 26379},
|
||||
]
|
||||
|
||||
## Second cluster that will host the queues, shared state, and actioncable
|
||||
## Second cluster that will host the persistent queues, shared state, and actioncable
|
||||
gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
|
||||
gitlab_rails['redis_queues_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
gitlab_rails['redis_shared_state_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
gitlab_rails['redis_actioncable_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
|
||||
# Gitaly
|
||||
|
|
@ -2063,30 +1816,30 @@ On each node perform the following:
|
|||
gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache'
|
||||
|
||||
gitlab_rails['redis_cache_sentinels'] = [
|
||||
{host: '10.6.0.71', port: 26379},
|
||||
{host: '10.6.0.72', port: 26379},
|
||||
{host: '10.6.0.73', port: 26379},
|
||||
{host: '10.6.0.51', port: 26379},
|
||||
{host: '10.6.0.52', port: 26379},
|
||||
{host: '10.6.0.53', port: 26379},
|
||||
]
|
||||
|
||||
## Second cluster that will host the queues, shared state, and actionable
|
||||
## Second cluster that will host the persistent queues, shared state, and actionable
|
||||
gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent'
|
||||
|
||||
gitlab_rails['redis_queues_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
gitlab_rails['redis_shared_state_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
gitlab_rails['redis_actioncable_sentinels'] = [
|
||||
{host: '10.6.0.81', port: 26379},
|
||||
{host: '10.6.0.82', port: 26379},
|
||||
{host: '10.6.0.83', port: 26379},
|
||||
{host: '10.6.0.61', port: 26379},
|
||||
{host: '10.6.0.62', port: 26379},
|
||||
{host: '10.6.0.63', port: 26379},
|
||||
]
|
||||
|
||||
# Set the network addresses that the exporters used for monitoring will listen on
|
||||
|
|
@ -2158,7 +1911,7 @@ On each node perform the following:
|
|||
|
||||
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
|
||||
|
||||
1. If you're [using NFS](#configure-nfs-optional):
|
||||
1. If you're [using NFS](#configure-nfs):
|
||||
1. If necessary, install the NFS client utility packages using the following
|
||||
commands:
|
||||
|
||||
|
|
@ -2300,7 +2053,7 @@ To configure the Monitoring node:
|
|||
## Configure the object storage
|
||||
|
||||
GitLab supports using an object storage service for holding numerous types of data.
|
||||
It's recommended over [NFS](#configure-nfs-optional) and in general it's better
|
||||
It's recommended over [NFS](#configure-nfs) and in general it's better
|
||||
in larger setups as object storage is typically much more performant, reliable,
|
||||
and scalable.
|
||||
|
||||
|
|
@ -2370,7 +2123,7 @@ cluster alongside your instance, read how to
|
|||
</a>
|
||||
</div>
|
||||
|
||||
## Configure NFS (optional)
|
||||
## Configure NFS
|
||||
|
||||
[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly)
|
||||
are recommended over NFS wherever possible for improved performance. If you intend
|
||||
|
|
@ -2438,10 +2191,8 @@ services where applicable):
|
|||
| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` |
|
||||
| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
|
||||
| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` |
|
||||
| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
|
||||
| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
|
||||
| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` |
|
||||
| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` |
|
||||
| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
|
||||
| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` |
|
||||
| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` |
|
||||
| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` |
|
||||
| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` |
|
||||
|
|
@ -2497,11 +2248,6 @@ card "Database" as database {
|
|||
card "redis" as redis {
|
||||
collections "**Redis Persistent** x3" as redis_persistent #FF6347
|
||||
collections "**Redis Cache** x3" as redis_cache #FF6347
|
||||
collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347
|
||||
collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347
|
||||
|
||||
redis_persistent <.[#FF6347]- redis_persistent_sentinel
|
||||
redis_cache <.[#FF6347]- redis_cache_sentinel
|
||||
}
|
||||
|
||||
cloud "**Object Storage**" as object_storage #white
|
||||
|
|
|
|||
|
|
@ -301,7 +301,7 @@ An alternative to `File` type variables is to:
|
|||
|
||||
```shell
|
||||
# Read certificate stored in $KUBE_CA_PEM variable and save it in a new file
|
||||
echo "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem"
|
||||
cat "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem"
|
||||
# Pass the newly created file to kubectl
|
||||
kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$(pwd)/kube.ca.pem"
|
||||
```
|
||||
|
|
|
|||
|
|
@ -10,9 +10,10 @@ type: howto, reference
|
|||
Deploy keys allow read-only or read-write access to your
|
||||
repositories by importing an SSH public key into your GitLab instance.
|
||||
|
||||
This is useful, for example, for cloning repositories to your Continuous
|
||||
Integration (CI) server. By using deploy keys, you don't have to set up a
|
||||
fake user account.
|
||||
Deploy keys streamline interactions between your GitLab repository and another
|
||||
machine. For example, setting up a deploy key allows secure cloning of your
|
||||
repositories to a Continuous Integration (CI) server without setting up a fake
|
||||
user account.
|
||||
|
||||
There are two types of deploy keys:
|
||||
|
||||
|
|
@ -63,11 +64,12 @@ help you access a repository, but there are some notables differences between th
|
|||
- Deploy keys are shareable between projects that are not related or don't even
|
||||
belong to the same group. Deploy tokens belong to either a project or
|
||||
[a group](../deploy_tokens/index.md#group-deploy-token).
|
||||
- A deploy key is an SSH key you need to generate yourself on your machine. A deploy
|
||||
token is generated by your GitLab instance, and is provided to users only once
|
||||
(at creation time).
|
||||
- A deploy key is valid as long as it's registered and enabled. Deploy tokens can
|
||||
be time-sensitive, as you can control their validity by setting an expiration date to them.
|
||||
- A deploy key is an SSH key you generate on the **remote machine**. A deploy
|
||||
token, on the other hand, is generated by your **GitLab instance**, and is
|
||||
provided to users only once (at creation time).
|
||||
- A deploy key is valid as long as it's registered and enabled. Deploy tokens
|
||||
can be time-sensitive, as you can control their validity by setting an
|
||||
expiration date to them.
|
||||
- You can't log in to a registry with deploy keys, or perform read / write operations
|
||||
on it, but this [is possible with deploy tokens](../deploy_tokens/index.md#gitlab-deploy-token).
|
||||
- You need an SSH key pair to use deploy keys, but not deploy tokens.
|
||||
|
|
@ -115,9 +117,9 @@ project, and if you then update the access level for this key from `read-only` t
|
|||
|
||||
### Public deploy keys
|
||||
|
||||
Public deploy keys allow `read-only` or `read-write`
|
||||
access to any repository in your GitLab instance. This is useful for integrating
|
||||
repositories to secure, shared services, such as CI/CD.
|
||||
Public deploy keys allow `read-only` or `read-write` access to any repository in
|
||||
your GitLab instance. This allows for the integration of your repositories to
|
||||
secure, shared services, such as CI/CD.
|
||||
|
||||
Instance administrators can add public deploy keys:
|
||||
|
||||
|
|
@ -125,32 +127,37 @@ Instance administrators can add public deploy keys:
|
|||
1. On the left sidebar, select **Deploy Keys**.
|
||||
1. Select **New deploy key**.
|
||||
|
||||
Make sure your new key has a meaningful title, as it is the primary way for project
|
||||
maintainers and owners to identify the correct public deploy key to add. For example,
|
||||
if the key gives access to a SaaS CI/CD instance, use the name of that service
|
||||
in the key name if that is all the key is used for.
|
||||
Make sure your new key has a meaningful title. This title is the primary
|
||||
way for project maintainers and owners to identify the correct public deploy key
|
||||
to add to a repository or project. For example, the key you set up might be
|
||||
intended to give access to a SaaS CI/CD instance. In this case use the name of
|
||||
that service in the key title if that is all the key is used for.
|
||||
|
||||
|
||||
|
||||
After adding a key, it's available to any shared systems. Project maintainers
|
||||
or higher can [authorize a public deploy key](#project-deploy-keys) to start using it with the project.
|
||||
After adding a key, it's available to any shared system. Users with a maintainer role or
|
||||
higher can [authorize a public deploy key](#project-deploy-keys) to start using
|
||||
it with the project.
|
||||
|
||||
NOTE:
|
||||
The **Publicly accessible deploy keys** tab within Project's CI/CD settings only appears
|
||||
if there is at least one Public deploy key configured.
|
||||
The **Publicly accessible deploy keys** tab in a Project's CI/CD
|
||||
settings only appears if there is at least one Public deploy key configured.
|
||||
|
||||
Public deploy keys can provide greater security compared to project deploy keys, as
|
||||
the administrator of the target integrated system is the only one who needs to know the key value,
|
||||
or configure it.
|
||||
Public deploy keys can provide greater security compared to project deploy keys.
|
||||
This is because the administrator of the target integrated system is the only
|
||||
entity who needs to know or configure the key value.
|
||||
|
||||
When creating a Public deploy key, determine whether or not it can be defined for
|
||||
very narrow usage, such as just a specific service, or if it needs to be defined for
|
||||
broader usage, such as full `read-write` access for all services.
|
||||
When creating a Public deploy key, consider what scope and permissions are
|
||||
required for it across the entire GitLab instance. For very narrow usage, such
|
||||
as a single specific service, a `read-only` deploy key tied to this service is
|
||||
best. If the service entails broader usage across the instance, a
|
||||
deploy key with full `read-write` access is more appropriate.
|
||||
|
||||
WARNING:
|
||||
Adding a public deploy key does not immediately expose any repository to it. Public
|
||||
deploy keys enable access from other systems, but access is not given to any project
|
||||
until a project maintainer chooses to make use of it.
|
||||
Adding a public deploy key **does not** immediately expose any repository
|
||||
to the remote machine. Access to a project is only given when a project
|
||||
maintainer chooses to make use of a deploy key in the project's
|
||||
configuration.
|
||||
|
||||
## How to disable deploy keys
|
||||
|
||||
|
|
@ -162,13 +169,17 @@ can remove or disable a deploy key for a project repository:
|
|||
1. Select the **{remove}** or **{cancel}** button.
|
||||
|
||||
NOTE:
|
||||
If anything relies on the removed deploy key, it will stop working once removed.
|
||||
Any service that relies on a deploy key stops working after that key is removed.
|
||||
|
||||
If the key is **publicly accessible**, it will be removed from the project, but still available under **Publicly accessible deploy keys**.
|
||||
If the key is **publicly accessible**, it is removed from the project, but can
|
||||
still be found under **Publicly accessible deploy keys**.
|
||||
|
||||
If the key is **privately accessible** and only in use by this project, it will deleted.
|
||||
If the key is **privately accessible** and only in use by this project, it is
|
||||
deleted entirely from GitLab on removal.
|
||||
|
||||
If the key is **privately accessible** and in use by other projects, it will be removed from the project, but still available under **Privately accessible deploy keys**.
|
||||
If the key is **privately accessible** and also in use by other projects, it is
|
||||
removed from the project, but still available under **Privately accessible
|
||||
deploy keys**.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ with financial transactions.
|
|||
|
||||
<!-- vale gitlab.Spelling = NO -->
|
||||
|
||||
Now we have a different picture. [According to Josh Aas](https://letsencrypt.org/2015/10/29/phishing-and-malware.html), Executive Director at [ISRG](https://en.wikipedia.org/wiki/Internet_Security_Research_Group):
|
||||
Now we have a different picture. [According to Josh Aas](https://letsencrypt.org/2015/10/29/phishing-and-malware.html), Executive Director at [Internet Security Research Group (ISRG)](https://en.wikipedia.org/wiki/Internet_Security_Research_Group):
|
||||
|
||||
<!-- vale gitlab.rulename = YES -->
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue