Remade documentation for Redis HA with Omnibus
This commit is contained in:
parent
494c2785fd
commit
1e4f867446
|
@ -1,9 +1,14 @@
|
|||
# Configuring Redis for GitLab HA
|
||||
|
||||
You can choose to install and manage Redis yourself, or you can use the one
|
||||
High Availability with Redis is possible using a **Master** x **Slave**
|
||||
topology with **Sentinel** service to watch and automatically start
|
||||
failover proceedings.
|
||||
|
||||
You can choose to install and manage Redis and Sentinel yourself, use
|
||||
a hosted, managed clouse solution or you can use or you can use the one
|
||||
that comes bundled with Omnibus GitLab packages.
|
||||
|
||||
> **Note:** Redis does not require authentication by default. See
|
||||
> **Note:** Redis requires authentication for High Availability. See
|
||||
[Redis Security](http://redis.io/topics/security) documentation for more
|
||||
information. We recommend using a combination of a Redis password and tight
|
||||
firewall rules to secure your Redis service.
|
||||
|
@ -12,20 +17,26 @@ that comes bundled with Omnibus GitLab packages.
|
|||
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
||||
**Table of Contents**
|
||||
|
||||
- [Configure your own Redis server](#configure-your-own-redis-server)
|
||||
- [Configure Redis using Omnibus](#configure-redis-using-omnibus)
|
||||
- [Experimental Redis Sentinel support](#experimental-redis-sentinel-support)
|
||||
- [Redis Sentinel support](#redis-sentinel-support)
|
||||
- [Using an external Redis server](#using-an-external-redis-server)
|
||||
- [High Availability with Sentinel](#high-availability-with-sentinel)
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Redis setup](#redis-setup)
|
||||
- [Existing single-machine installation](#existing-single-machine-installation)
|
||||
- [Omnibus packages](#omnibus-packages)
|
||||
- [Configuring Sentinel](#configuring-sentinel)
|
||||
- [How sentinel handles a failover](#how-sentinel-handles-a-failover)
|
||||
- [Sentinel setup](#sentinel-setup)
|
||||
- [Experimental Redis Sentinel support](#experimental-redis-sentinel-support)
|
||||
- [Sentinel setup](#sentinel-setup)
|
||||
- [Recommended setup](#recommended-setup)
|
||||
- [Configuring instances using Omnibus](#configuring-instances-using-omnibus)
|
||||
- [Existing single-machine installation](#existing-single-machine-installation)
|
||||
- [Configuring Master Redis instance](#configuring-master-redis-instance)
|
||||
- [Configuring Slave Redis instances](#configuring-slave-redis-instances)
|
||||
- [Configuring Sentinel instances](#configuring-sentinel-instances)
|
||||
- [Community Edition](#community-edition)
|
||||
- [Enterprise Edition](#enterprise-edition)
|
||||
- [GitLab setup](#gitlab-setup)
|
||||
- [Example Configurations](#example-configurations)
|
||||
- [Configuration for Redis Master](#configuration-for-redis-master)
|
||||
- [Configuration for Redis Slave](#configuration-for-redis-slave)
|
||||
- [Configuration for Sentinel (EE only)](#configuration-for-sentinel-ee-only)
|
||||
- [Control running services](#control-running-services)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Redis replication](#redis-replication)
|
||||
- [Sentinel](#sentinel)
|
||||
|
@ -33,75 +44,57 @@ that comes bundled with Omnibus GitLab packages.
|
|||
|
||||
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
||||
|
||||
## Configure your own Redis server
|
||||
## Using an external Redis server
|
||||
|
||||
If you're hosting GitLab on a cloud provider, you can optionally use a
|
||||
managed service for Redis. For example, AWS offers a managed ElastiCache service
|
||||
that runs Redis.
|
||||
|
||||
## Configure Redis using Omnibus
|
||||
Managed services can provide High Availability using their own proprietary
|
||||
technology and provide a transparent proxy, which means that GitLab doesn't
|
||||
need any additional change, or will use Sentinel and manage it for you.
|
||||
|
||||
If you don't want to bother setting up your own Redis server, you can use the
|
||||
one bundled with Omnibus. In this case, you should disable all services except
|
||||
Redis.
|
||||
If your provider, uses Sentinel method, see [GitLab Setup](#gitlab-setup)
|
||||
to understant where you need to provide the list of servers and credentials.
|
||||
|
||||
1. Download/install Omnibus GitLab using **steps 1 and 2** from
|
||||
[GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
|
||||
steps on the download page.
|
||||
1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
|
||||
Be sure to change the `external_url` to match your eventual GitLab front-end
|
||||
URL:
|
||||
If you want to setup Redis by yourself, without using Omnibus, you can
|
||||
read our documentation: [Configuring Redis for GitLab HA (Source Install)](redis_source.md)
|
||||
|
||||
```ruby
|
||||
external_url 'https://gitlab.example.com'
|
||||
## High Availability with Sentinel
|
||||
|
||||
# Disable all services except Redis
|
||||
redis['enable'] = true
|
||||
bootstrap['enable'] = false
|
||||
nginx['enable'] = false
|
||||
postgresql['enable'] = false
|
||||
gitlab_rails['enable'] = false
|
||||
> Since GitLab `8.11`, you can configure a list of Redis Sentinel servers that
|
||||
will monitor a group of Redis servers to provide failover support.
|
||||
|
||||
# Redis configuration
|
||||
redis['port'] = 6379
|
||||
redis['bind'] = '0.0.0.0'
|
||||
redis['password'] = 'redis-password-goes-here'
|
||||
```
|
||||
> With GitLab `8.14`, we bundled Redis Sentinel as part of Omnibus package and
|
||||
improved the way you use and configure it.
|
||||
|
||||
1. Run `sudo gitlab-ctl reconfigure` to install and configure Redis.
|
||||
High Availability with Redis requires a few things:
|
||||
|
||||
> **Note**: This `reconfigure` step will result in some errors.
|
||||
That's OK - don't be alarmed.
|
||||
|
||||
1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations
|
||||
from running on upgrade. Only the primary GitLab application server should
|
||||
handle migrations.
|
||||
|
||||
## Experimental Redis Sentinel support
|
||||
|
||||
> Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11.
|
||||
Starting with 8.14, Redis Sentinel is no longer experimental.
|
||||
If you used with versions `< 8.14` before, please check the updated
|
||||
documentation below.
|
||||
|
||||
## Redis Sentinel support
|
||||
|
||||
Since GitLab 8.11, you can configure a list of Redis Sentinel servers that
|
||||
will monitor a group of Redis servers to provide you with a standard failover
|
||||
support.
|
||||
|
||||
To get a better understanding on how to correctly setup Sentinel, please read
|
||||
the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
|
||||
failing to configure it correctly can lead to data loss.
|
||||
- Multiple Redis instances
|
||||
- Run Redis in a **Master** x **Slave** topology
|
||||
- Multiple Sentinel instances
|
||||
- Application support and visiblity to all Sentinel and Redis instances
|
||||
|
||||
Redis Sentinel can handle the most important tasks in a HA environment to help
|
||||
keep servers online with minimal to no downtime:
|
||||
|
||||
- Monitors master and slave instances to see if they are available
|
||||
- Promote a slave to master when the master fails.
|
||||
- Demote a master to slave when failed master comes back online (to prevent
|
||||
- Monitors **Master** and **Slaves** instances to see if they are available
|
||||
- Promote a **Slave** to **Master** when the **Master** fails.
|
||||
- Demote a **Master** to **Slave** when failed **Master** comes back online (to prevent
|
||||
data-partitioning).
|
||||
- Can be queried by clients to always connect to the correct master server.
|
||||
- Can be queried by clients to always connect to the current **Master** server.
|
||||
|
||||
When a **Master** fails to respond, it's the client responsability to handle timeout
|
||||
and reconnect (querying a **Sentinel** for a new **Master**).
|
||||
|
||||
To get a better understanding on how to correctly setup Sentinel, please read
|
||||
the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
|
||||
failing to configure it correctly can lead to data loss, or can bring your
|
||||
whole cluster down, invalidating the failover effort.
|
||||
|
||||
This documentation will provide you with a minimal and a recommended topology
|
||||
that can resist to some levels of failure. Usually the more Redis and Sentinel
|
||||
instances you have provisioned, the better will be your availability.
|
||||
|
||||
The configuration consists of three parts:
|
||||
|
||||
|
@ -112,44 +105,191 @@ The configuration consists of three parts:
|
|||
### Prerequisites
|
||||
|
||||
You need at least `3` independent machines: physical, or VMs running into
|
||||
distinct physical machines.
|
||||
distinct physical machines. They must be believed to fail in an
|
||||
independent way.
|
||||
|
||||
If you fail to provision the machines in that specific way, any issue with
|
||||
the shared environment can bring your entire setup down.
|
||||
|
||||
Read carefully how to configure those components below.
|
||||
You also need to take in consideration the underlying network topology,
|
||||
making sure you have redundant connectivity between Redis / Sentinel and
|
||||
GitLab instances, otherwhise the networks will become a single point of
|
||||
failure.
|
||||
|
||||
Read carefully how to configure the components below.
|
||||
|
||||
### Redis setup
|
||||
|
||||
You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they need to
|
||||
be each in a independent machine (see explanation above).
|
||||
You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they
|
||||
need to be each in a independent machine (see explanation above).
|
||||
|
||||
They should be configured the same way and with similar server specs, as
|
||||
in a failover situation, any `Slave` can be elected as the new `Master` by
|
||||
You can have additional Redis nodes, that will help survive a situation
|
||||
where more nodes goes down. Whenever there is only `2` nodes online, a failover
|
||||
will not be initiated.
|
||||
|
||||
As an example, if you have `6` Redis nodes, a maximum of `3` can be
|
||||
simultaneously down.
|
||||
|
||||
Please note that there are different requirements for Sentinel nodes.
|
||||
If you host them in the same Redis machines, you may need to take
|
||||
that restrictions into consideration when calculating the ammount of
|
||||
nodes to be provisioned. See [Sentinel setup](#sentinel-setup)
|
||||
documentation for more information.
|
||||
|
||||
All Redis nodes should be configured the same way and with similar server specs, as
|
||||
in a failover situation, any **Slave** can be promoted as the new **Master** by
|
||||
the Sentinel servers.
|
||||
|
||||
With Sentinel, you must define a password to protect the access as both
|
||||
Sentinel instances and other redis instances should be able to talk to
|
||||
The replication requires authentication, so you need to define a password to
|
||||
protect all Redis nodes and the Sentinels. They will all share the same
|
||||
password, and all instances must be able to talk to
|
||||
each other over the network.
|
||||
|
||||
You'll need to define both `requirepass` and `masterauth` in all
|
||||
nodes. At any time during a failover the Sentinels can reconfigure a node
|
||||
and change it's status from `Master` to `Slave` and vice versa.
|
||||
Redis nodes will need the same password defined in `redis['password']` and
|
||||
`redis['master_password']`, no matter if **Master** or **Slave**. At any time
|
||||
during a failover the Sentinels can reconfigure a node and change it's status
|
||||
from **Master** to **Slave** and vice versa.
|
||||
|
||||
Initial `Slave` nodes require an additional `slaveof` setting in `redis.conf`
|
||||
pointing to the initial `Master`.
|
||||
Initial **Slave** nodes requires `redis['master']` defined to `false` and
|
||||
`redis['master_ip']` pointing to the initial **Master**. If you use the
|
||||
simplified configuration by enabling `redis_slave_role['enable']`, you
|
||||
just need to fill in the `redis['master_ip']`.
|
||||
|
||||
#### Existing single-machine installation
|
||||
This values doesn't have to be changed again in `/etc/gitlab/gitlab.rb` after
|
||||
a failover, as the nodes will be managed by the Sentinels, and even after a
|
||||
`gitlab-ctl reconfigure`, they will get their configuration restored by
|
||||
the same Sentinels.
|
||||
|
||||
### Experimental Redis Sentinel support
|
||||
|
||||
> Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11.
|
||||
Starting with 8.14, Redis Sentinel is no longer experimental.
|
||||
If you used with versions `< 8.14` before, please check the updated
|
||||
documentation here.
|
||||
|
||||
### Sentinel setup
|
||||
|
||||
Sentinels watches both other sentinels and Redis nodes. Whenever a Sentinel
|
||||
detects that a Redis node is not responding, it will announce that to the
|
||||
other sentinels. You have to reach the **quorum**, the minimum ammount of
|
||||
sentinels that agrees that a node is down, to be able to start a failover.
|
||||
|
||||
Whenver the **quorum** is met, you need the **majority** of all known
|
||||
Sentinel nodes to be available and reachable, to elect the Sentinel **leader**
|
||||
who will take all the decisions to restore the service availability by:
|
||||
|
||||
- Promoting a new **Master**
|
||||
- Reconfiguring the other **Slaves** and make them point to the new **Master**
|
||||
- Announce the new **Master** to every other Sentinel peer
|
||||
- Reconfigure the old **Master** and demote to **Slave** when it comes back online
|
||||
|
||||
You must have at least `3` Redis Sentinel servers, and they need to
|
||||
be each in a independent machine (that are believed to fail independently).
|
||||
|
||||
You can configure them in the same machines where you've configured the other
|
||||
Redis servers, but understand that if a whole node goes down, you loose both
|
||||
a Sentinel and a Redis instance.
|
||||
|
||||
The number of sentinels should ideally always be an **odd** number, for the
|
||||
consensus algorithm to be effective in the case of a failure.
|
||||
|
||||
In a `3` nodes topology, you can only afford `1` Sentinel node going down.
|
||||
Whenever the **majority** of the Sentinels goes down, the network partition
|
||||
protection prevents destructive actions and a failover **will not be started**.
|
||||
|
||||
Here are some examples:
|
||||
|
||||
- With `5` or `6` sentinels, a maximum of `2` can go down for a failover begin.
|
||||
- With `7` sentinels, a maximum of `3` nodes can go down.
|
||||
|
||||
The **Leader** election can sometimes fail the voting round when **consensus**,
|
||||
is not achieved (see the odd number of nodes requirement above). In that case,
|
||||
a new attempt will be made after the amount of time defined in
|
||||
`sentinel['failover_timeout']` (in milliseconds).
|
||||
|
||||
The `failover_time` variable have a lot of different usages, according to
|
||||
official documentation:
|
||||
|
||||
- The time needed to re-start a failover after a previous failover was
|
||||
already tried against the same master by a given Sentinel, is two
|
||||
times the failover timeout.
|
||||
|
||||
- The time needed for a slave replicating to a wrong master according
|
||||
to a Sentinel current configuration, to be forced to replicate
|
||||
with the right master, is exactly the failover timeout (counting since
|
||||
the moment a Sentinel detected the misconfiguration).
|
||||
|
||||
- The time needed to cancel a failover that is already in progress but
|
||||
did not produced any configuration change (SLAVEOF NO ONE yet not
|
||||
acknowledged by the promoted slave).
|
||||
|
||||
- The maximum time a failover in progress waits for all the slaves to be
|
||||
reconfigured as slaves of the new master. However even after this time
|
||||
the slaves will be reconfigured by the Sentinels anyway, but not with
|
||||
the exact parallel-syncs progression as specified.
|
||||
|
||||
### Recommended setup
|
||||
|
||||
For a minimal setup, you will install the Omnibus GitLab package in `3`
|
||||
independent machines, both with **Redis** and **Sentinel**:
|
||||
|
||||
- Redis Master + Sentinel
|
||||
- Redis Slave + Sentinel
|
||||
- Redis Slave + Sentinel
|
||||
|
||||
Make sure you've read [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup)
|
||||
before, to understant how and why the ammount of nodes came from.
|
||||
|
||||
For a recommended setup, that can resist more failures, you will install
|
||||
the Omnibus GitLab package in `5` independent machines, both with
|
||||
**Redis** and **Sentinel**:
|
||||
|
||||
- Redis Master + Sentinel
|
||||
- Redis Slave + Sentinel
|
||||
- Redis Slave + Sentinel
|
||||
- Redis Slave + Sentinel
|
||||
- Redis Slave + Sentinel
|
||||
|
||||
## Configuring instances using Omnibus
|
||||
|
||||
This is a summary of what are we going to do:
|
||||
|
||||
1. Provision the required number of instances specified previously
|
||||
- You can opt to install Redis and Sentinel in the same machine or each in
|
||||
independent ones.
|
||||
- Don't install Redis and Sentinel in the same machines your GitLab instance
|
||||
is running on.
|
||||
- All machines must be able to talk to each other and accept incomming
|
||||
connection over Redis (`6379`) and Sentinel (`26379`) ports.
|
||||
- GitLab machines must be able to access these machines and with the same
|
||||
permissions.
|
||||
- Protected them from indiscriminated access from external networks (Internet),
|
||||
to harden the security.
|
||||
|
||||
1. Download/install Omnibus GitLab using **steps 1 and 2** from
|
||||
[GitLab downloads](https://about.gitlab.com/downloads) in each node.
|
||||
- Do not complete other steps on the download page.
|
||||
- Make sure you select the correct Omnibus package, with the same version
|
||||
and type (Community, Enterprise editions) of your current install.
|
||||
|
||||
1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations
|
||||
from running on upgrade. Only the primary GitLab application server should
|
||||
handle migrations.
|
||||
|
||||
1. Create/edit `/etc/gitlab/gitlab.rb` and make the changes based on the
|
||||
[Example Configurations](#example-configurations).
|
||||
|
||||
### Existing single-machine installation
|
||||
|
||||
If you already have a single-machine GitLab install running, you will need to
|
||||
replicate from this machine first, before de-activating the Redis instance
|
||||
inside it.
|
||||
|
||||
Your single-machine install will be the initial `Master`, and the `3` others
|
||||
should be configured as `Slave` pointing to this machine.
|
||||
Your single-machine install will be the initial **Master**, and the `3` others
|
||||
should be configured as **Slave** pointing to this machine.
|
||||
|
||||
After replication catchs-up, you will need to stop services in the
|
||||
single-machine install, to rotate the `Master` to one of the new nodes.
|
||||
single-machine install, to rotate the **Master** to one of the new nodes.
|
||||
|
||||
Make the required changes in configuration and restart the new nodes again.
|
||||
|
||||
|
@ -159,130 +299,209 @@ To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`:
|
|||
redis['enable'] = false
|
||||
```
|
||||
|
||||
#### Omnibus packages
|
||||
If you fail to replicate first, you may loose data (unprocessed background jobs).
|
||||
|
||||
You need to install the Omnibus GitLab package in `3` independent machines.
|
||||
### Configuring Master Redis instance
|
||||
|
||||
**Configuring Master Redis instance**
|
||||
You will need to configure the following in `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
You will need to configure the following:
|
||||
1. Define `redis_master_role['enable']` to `true`, to disable other services
|
||||
in the machine (you can still enable Sentinel)
|
||||
|
||||
1. Define a `redis['bind']` address pointing to a local IP that your other machines
|
||||
can reach you. If you really need to bind to an external acessible IP, make
|
||||
can reach you.
|
||||
- If you really need to bind to an external acessible IP, make
|
||||
sure you add extra firewall rules to prevent unauthorized access.
|
||||
- You can also set bind to `0.0.0.0` which listen in all interfaces.
|
||||
|
||||
1. Define a `redis['port']` so redis can listen for TCP requests which will
|
||||
allow other machines to connect to it.
|
||||
1. Set up a password authentication with `redis['master_password']` (use the same
|
||||
password in all nodes).
|
||||
|
||||
In `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```ruby
|
||||
## Redis TCP support (will disable UNIX socket transport)
|
||||
redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
|
||||
redis['port'] = 6379
|
||||
redis['password'] = 'redis-password-goes-here'
|
||||
redis['master_password'] = 'redis-password-goes-here'
|
||||
```
|
||||
1. Set up a password authentication with `redis['password']` and
|
||||
`redis['master_password']` (use the same password in all nodes).
|
||||
|
||||
Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure`
|
||||
|
||||
**Configuring Slave Redis instances**
|
||||
### Configuring Slave Redis instances
|
||||
|
||||
You need to make the same changes listed for the `Master` instance,
|
||||
with an additional `Slave` section as in the example below:
|
||||
You will need to configure the following in `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```ruby
|
||||
redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
|
||||
redis['port'] = 6379
|
||||
redis['password'] = 'redis-password-goes-here'
|
||||
redis['master_password'] = 'redis-password-goes-here'
|
||||
1. Define `redis_slaves_role['enable']` to `true`, to disable other services
|
||||
in the machine (you can still enable Sentinel)
|
||||
- This will also set automatically `redis['master'] = false`.
|
||||
|
||||
## Slave redis instance
|
||||
redis['master'] = false
|
||||
redis['master_ip'] = '10.10.10.10' # IP of master Redis server
|
||||
redis['master_port'] = 6379 # Port of master Redis server
|
||||
```
|
||||
1. Define a `redis['bind']` address pointing to a local IP that your other machines
|
||||
can reach you.
|
||||
- If you really need to bind to an external acessible IP, make
|
||||
sure you add extra firewall rules to prevent unauthorized access.
|
||||
- You can also set bind to `0.0.0.0` which listen in all interfaces.
|
||||
|
||||
Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure`
|
||||
1. Define a `redis['port']` so redis can listen for TCP requests which will
|
||||
allow other machines to connect to it.
|
||||
|
||||
---
|
||||
1. Set up a password authentication with `redis['password']` and
|
||||
`redis['master_password']` (use the same password in all nodes).
|
||||
|
||||
1. Define `redis['master_ip']` with the IP of the **Master** Redis.
|
||||
|
||||
1. Define `redis['master_port']` with the port of the **Master** Redis (default to `6379`).
|
||||
|
||||
### Configuring Sentinel instances
|
||||
|
||||
Now that the Redis servers are all set up, let's configure the Sentinel
|
||||
servers.
|
||||
|
||||
If you are not sure if your Redis servers are working and replicating
|
||||
correctly, please read the [Troubleshooting Replication](#troubleshooting-replication)
|
||||
correctly, please read the [Troubleshooting Replication](#troubleshooting-replication)
|
||||
and fix it before proceeding with Sentinel setup.
|
||||
|
||||
### Configuring Sentinel
|
||||
|
||||
You must have at least `3` Redis Sentinel servers, and they need to
|
||||
be each in a independent machine. You can configure them in the same
|
||||
machines where you've configured the other Redis servers.
|
||||
|
||||
This number is required for the consensus algorithm to be effective
|
||||
in the case of a failure. **You should always have an `odd` number
|
||||
of Sentinel nodes provisioned**.
|
||||
|
||||
#### How sentinel handles a failover
|
||||
|
||||
If (`quorum` value of) Sentinels agree the fact the `master` is not reachable,
|
||||
Sentinels will try to elect a temporary `Leader`. The **Majority** of the
|
||||
Sentinels must agree to start a failover.
|
||||
|
||||
If you don't have the **Majority** of the Sentinels online (for example if you
|
||||
are under a network partitioning), a failover **will not be started**.
|
||||
|
||||
For example, for a cluster of `3` Sentinels, at least `2` must agree on a
|
||||
`Leader`. If you have total of `5` at least `3` must agree on a `Leader`.
|
||||
|
||||
The `quorum` is only used to detect failure, not to elect the `Leader`.
|
||||
|
||||
Official [Sentinel documentation](http://redis.io/topics/sentinel#example-sentinel-deployments)
|
||||
also lists different network topologies and warns againts situations like
|
||||
network partition and how it can affect the state of the HA solution. Make
|
||||
sure you read it carefully and understand the implications in your current
|
||||
setup.
|
||||
|
||||
GitLab Enterprise Edition provides [automated way to setup and run](#sentinel-setup-ee-only) the Sentinel daemon.
|
||||
|
||||
#### Sentinel setup
|
||||
|
||||
##### Community Edition
|
||||
|
||||
With GitLab Community Edition, you need to install, configure, execute and
|
||||
monitor Sentinel from source. Omnibus GitLab Community Edition package does
|
||||
not support Sentinel configuration.
|
||||
|
||||
See documentation for Source Install [here](redis_source.md).
|
||||
See [documentation for Source Install](redis_source.md).
|
||||
|
||||
##### Enterprise Edition
|
||||
|
||||
To setup sentinel, edit `/etc/gitlab/gitlab.rb` file:
|
||||
With GitLab Enterprise Edition, you can use Omnibus package to setup multiple
|
||||
machines with Sentinel daemon.
|
||||
|
||||
See [example configuration](#configuration-for-sentinel-ee-only) below.
|
||||
|
||||
### GitLab setup
|
||||
|
||||
The final part is to inform the main GitLab application server of the Redis
|
||||
Sentinels servers and authentication credentials.
|
||||
|
||||
You can enable or disable Sentinel support at any time in new or existing
|
||||
installations. From the GitLab application perspective, all it requires is
|
||||
the correct credentials for the Sentinel nodes.
|
||||
|
||||
While it doesn't require a list of all Sentinel nodes, in case of a failure,
|
||||
it needs to access at least one of listeds.
|
||||
|
||||
>**Note:**
|
||||
The following steps should be performed in the [GitLab application server](gitlab.md)
|
||||
which ideally should not have Redis or Sentinels in the same machine for a HA setup.
|
||||
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:
|
||||
|
||||
- `redis['master_name']` - this is the `master-group-name` from sentinel (default: `gitlab-redis`)
|
||||
- `redis['master_password']` - the same password you've defined before for Redis and Sentinels
|
||||
- `gitlab_rails['redis_sentinels']` - a list of sentinels with `host` and `port`
|
||||
|
||||
1. [Reconfigure] GitLab for the changes to take effect.
|
||||
|
||||
See [example configuration](#configuration-for-gitlab) below.
|
||||
|
||||
## Example Configurations
|
||||
|
||||
In this example we consider that all servers have an internal network
|
||||
interface with IPs in the `10.0.0.x` range, and that they can connect
|
||||
to each other using these IPs.
|
||||
|
||||
In a real world usage, you would also setup firewall rules to prevent
|
||||
unauthorized access from other machines, and block traffic from the
|
||||
outside (Internet).
|
||||
|
||||
We will use the same `3` nodes with **Redis** + **Sentinel** topology
|
||||
discussed in [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup)
|
||||
documentation.
|
||||
|
||||
Here is a list and description of each **machine** and the assigned **IP**:
|
||||
|
||||
* `10.0.0.1`: Redis Master + Sentinel 1
|
||||
* `10.0.0.2`: Redis Slave 1 + Sentinel 2
|
||||
* `10.0.0.2`: Redis Slave 2 + Sentinel 3
|
||||
|
||||
Please note that after the initial configuration, if a failover is initiated
|
||||
by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master**
|
||||
will change permanently (including in `redis.conf`) from one node to the other,
|
||||
until a new failover is initiated again.
|
||||
|
||||
The same thing will happen with `sentinel.conf` that will be overriten after the
|
||||
initial execution, after any new sentinel node starts watching the **Master**,
|
||||
or a failover promotes a different **Master** node.
|
||||
|
||||
### Configuration for Redis Master
|
||||
|
||||
**Example configation for Redis Master:**
|
||||
|
||||
In `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```ruby
|
||||
|
||||
## When you install Sentinel in a separate machine, you need to control which
|
||||
## other services will be running in it.
|
||||
## We've simplified the choice using special "roles" settings:
|
||||
|
||||
## Enabled Sentinel and Redis Master services
|
||||
redis_sentinel_role['enable'] = true
|
||||
redis_master_role['enable'] = true
|
||||
|
||||
## Enabled Sentinel and Redis Slave services
|
||||
redis_sentinel_role['enable'] = true
|
||||
redis['bind'] = '10.0.0.1'
|
||||
redis['port'] = 6379
|
||||
redis['password'] = 'redis-password-goes-here'
|
||||
redis['master_password'] = 'redis-password-goes-here'
|
||||
```
|
||||
Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure`
|
||||
|
||||
### Configuration for Redis Slave
|
||||
|
||||
**Example configation for Slave 1:**
|
||||
|
||||
In `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```ruby
|
||||
redis_slave_role['enable'] = true
|
||||
|
||||
## Configure Redis
|
||||
redis['bind'] = '10.0.0.2'
|
||||
redis['port'] = 6379
|
||||
redis['password'] = 'redis-password-goes-here'
|
||||
redis['master_password'] = 'redis-password-goes-here'
|
||||
|
||||
redis['master_ip'] = '10.0.0.1' # IP of master Redis server
|
||||
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
|
||||
```
|
||||
|
||||
Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure`
|
||||
|
||||
**Example configation for Slave 2:**
|
||||
|
||||
In `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```ruby
|
||||
redis_slave_role['enable'] = true
|
||||
|
||||
redis['bind'] = '10.0.0.3'
|
||||
redis['port'] = 6379
|
||||
redis['password'] = 'redis-password-goes-here'
|
||||
redis['master_password'] = 'redis-password-goes-here'
|
||||
|
||||
redis['master_ip'] = '10.0.0.1' # IP of master Redis server
|
||||
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
|
||||
```
|
||||
|
||||
Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure`
|
||||
|
||||
### Configuration for Sentinel (EE only)
|
||||
|
||||
Please note that some of the variables are already configured previously
|
||||
as they are required for Redis replication.
|
||||
|
||||
**Example configation for Sentinel 1:**
|
||||
|
||||
In `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```ruby
|
||||
redis_sentinel_role['enable'] = true
|
||||
|
||||
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
|
||||
redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance
|
||||
redis['master_port'] = 6379 # port of the initial master redis instance
|
||||
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
|
||||
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
|
||||
|
||||
## Configure Sentinel
|
||||
# sentinel['bind'] = '0.0.0.0' # bind to all interfaces, uncomment to specify an IP and bind to a single one
|
||||
sentinel['bind'] = '10.0.0.1'
|
||||
# sentinel['port'] = 26379 # uncomment to change default port
|
||||
|
||||
## Quorum must reflect the amount of voting sentinels it take to start a failover.
|
||||
|
@ -323,6 +542,116 @@ sentinel['quorum'] = 2
|
|||
# sentinel['failover_timeout'] = 60000
|
||||
```
|
||||
|
||||
**Example configation for Sentinel 2:**
|
||||
|
||||
In `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```ruby
|
||||
redis_sentinel_role['enable'] = true
|
||||
|
||||
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
|
||||
redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance
|
||||
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
|
||||
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
|
||||
|
||||
## Configure Sentinel
|
||||
sentinel['bind'] = '10.0.0.2'
|
||||
# sentinel['port'] = 26379 # uncomment to change default port
|
||||
|
||||
## Quorum must reflect the amount of voting sentinels it take to start a failover.
|
||||
## Value must NOT be greater then the ammount of sentinels.
|
||||
##
|
||||
## The quorum can be used to tune Sentinel in two ways:
|
||||
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
|
||||
## we deploy, we are basically making Sentinel more sensible to master failures,
|
||||
## triggering a failover as soon as even just a minority of Sentinels is no longer
|
||||
## able to talk with the master.
|
||||
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
|
||||
## making Sentinel able to failover only when there are a very large number (larger
|
||||
## than majority) of well connected Sentinels which agree about the master being down.s
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
## Consider unresponsive server down after x amount of ms.
|
||||
# sentinel['down_after_milliseconds'] = 10000
|
||||
|
||||
## Specifies the failover timeout in milliseconds. It is used in many ways:
|
||||
##
|
||||
## - The time needed to re-start a failover after a previous failover was
|
||||
## already tried against the same master by a given Sentinel, is two
|
||||
## times the failover timeout.
|
||||
##
|
||||
## - The time needed for a slave replicating to a wrong master according
|
||||
## to a Sentinel current configuration, to be forced to replicate
|
||||
## with the right master, is exactly the failover timeout (counting since
|
||||
## the moment a Sentinel detected the misconfiguration).
|
||||
##
|
||||
## - The time needed to cancel a failover that is already in progress but
|
||||
## did not produced any configuration change (SLAVEOF NO ONE yet not
|
||||
## acknowledged by the promoted slave).
|
||||
##
|
||||
## - The maximum time a failover in progress waits for all the slaves to be
|
||||
## reconfigured as slaves of the new master. However even after this time
|
||||
## the slaves will be reconfigured by the Sentinels anyway, but not with
|
||||
## the exact parallel-syncs progression as specified.
|
||||
# sentinel['failover_timeout'] = 60000
|
||||
```
|
||||
|
||||
**Example configation for Sentinel 3:**
|
||||
|
||||
In `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```ruby
|
||||
redis_sentinel_role['enable'] = true
|
||||
|
||||
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
|
||||
redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance
|
||||
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
|
||||
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
|
||||
|
||||
## Configure Sentinel
|
||||
sentinel['bind'] = '10.0.0.3'
|
||||
# sentinel['port'] = 26379 # uncomment to change default port
|
||||
|
||||
## Quorum must reflect the amount of voting sentinels it take to start a failover.
|
||||
## Value must NOT be greater then the ammount of sentinels.
|
||||
##
|
||||
## The quorum can be used to tune Sentinel in two ways:
|
||||
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
|
||||
## we deploy, we are basically making Sentinel more sensible to master failures,
|
||||
## triggering a failover as soon as even just a minority of Sentinels is no longer
|
||||
## able to talk with the master.
|
||||
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
|
||||
## making Sentinel able to failover only when there are a very large number (larger
|
||||
## than majority) of well connected Sentinels which agree about the master being down.s
|
||||
sentinel['quorum'] = 2
|
||||
|
||||
## Consider unresponsive server down after x amount of ms.
|
||||
# sentinel['down_after_milliseconds'] = 10000
|
||||
|
||||
## Specifies the failover timeout in milliseconds. It is used in many ways:
|
||||
##
|
||||
## - The time needed to re-start a failover after a previous failover was
|
||||
## already tried against the same master by a given Sentinel, is two
|
||||
## times the failover timeout.
|
||||
##
|
||||
## - The time needed for a slave replicating to a wrong master according
|
||||
## to a Sentinel current configuration, to be forced to replicate
|
||||
## with the right master, is exactly the failover timeout (counting since
|
||||
## the moment a Sentinel detected the misconfiguration).
|
||||
##
|
||||
## - The time needed to cancel a failover that is already in progress but
|
||||
## did not produced any configuration change (SLAVEOF NO ONE yet not
|
||||
## acknowledged by the promoted slave).
|
||||
##
|
||||
## - The maximum time a failover in progress waits for all the slaves to be
|
||||
## reconfigured as slaves of the new master. However even after this time
|
||||
## the slaves will be reconfigured by the Sentinels anyway, but not with
|
||||
## the exact parallel-syncs progression as specified.
|
||||
# sentinel['failover_timeout'] = 60000
|
||||
```
|
||||
|
||||
### Control running services
|
||||
|
||||
In the example above we've used `redis_sentinel_role` and `redis_master_role`
|
||||
which simplify the ammount of configuration changes.
|
||||
|
||||
|
@ -365,37 +694,6 @@ mailroom['enable'] = false
|
|||
# Redis Slave role also change this setting from default 'true' to 'false':
|
||||
redis['master'] = false
|
||||
```
|
||||
---
|
||||
|
||||
The final part is to inform the main GitLab application server of the Redis
|
||||
master and the new sentinels servers.
|
||||
|
||||
### GitLab setup
|
||||
|
||||
You can enable or disable Sentinel support at any time in new or existing
|
||||
installations. From the GitLab application perspective, all it requires is
|
||||
the correct credentials for the Sentinel nodes.
|
||||
|
||||
While it doesn't require a list of all Sentinel nodes, in case of a failure,
|
||||
it needs to access at one of listed ones.
|
||||
|
||||
>**Note:**
|
||||
The following steps should be performed in the [GitLab application server](gitlab.md)
|
||||
which ideally should not have Redis or Sentinels in the same machine for a HA setup.
|
||||
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:
|
||||
|
||||
```ruby
|
||||
redis['master_name'] = "gitlab-redis"
|
||||
redis['master_password'] = 'redis-password-goes-here'
|
||||
gitlab_rails['redis_sentinels'] = [
|
||||
{'host' => '10.10.10.1', 'port' => 26379},
|
||||
{'host' => '10.10.10.2', 'port' => 26379},
|
||||
{'host' => '10.10.10.3', 'port' => 26379}
|
||||
]
|
||||
```
|
||||
|
||||
1. [Reconfigure] GitLab for the changes to take effect.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
|
|
@ -153,7 +153,7 @@ outside (Internet).
|
|||
We will use the same `3` nodes with **Redis** + **Sentinel** topology
|
||||
discussed in the [Configuring Redis for GitLab HA](redis.md) documentation.
|
||||
|
||||
Here is a list and description of each **machine** and the assined **ip**:
|
||||
Here is a list and description of each **machine** and the assigned **IP**:
|
||||
|
||||
* `10.0.0.1`: Redis Master + Sentinel 1
|
||||
* `10.0.0.2`: Redis Slave 1 + Sentinel 2
|
||||
|
@ -170,7 +170,7 @@ or a failover promotes a different **Master** node.
|
|||
|
||||
### Configuring Redis Master
|
||||
|
||||
`redis.conf`:
|
||||
**Example configation for Redis Master - `redis.conf`:**
|
||||
|
||||
```conf
|
||||
bind 10.0.0.1
|
||||
|
@ -181,7 +181,7 @@ masterauth redis-password-goes-here
|
|||
|
||||
### Configuring Redis Slaves
|
||||
|
||||
**Slave 1 - `redis.conf`:**
|
||||
**Example configation for Slave 1 - `redis.conf`:**
|
||||
|
||||
```conf
|
||||
bind 10.0.0.2
|
||||
|
@ -193,7 +193,7 @@ masterauth redis-password-goes-here
|
|||
slaveof 10.0.0.1 6379
|
||||
```
|
||||
|
||||
**Slave 2 - `redis.conf`:**
|
||||
**Example configation for Slave 2 - `redis.conf`:**
|
||||
|
||||
```conf
|
||||
bind 10.0.0.3
|
||||
|
@ -211,7 +211,7 @@ For this example, **Sentinel 1** will be configured in the same machine as the
|
|||
**Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the
|
||||
**Slave 1** and **Slave 2** respectively.
|
||||
|
||||
Sentinel 1 - `sentinel.conf`
|
||||
**Example configation for Sentinel 1 - `sentinel.conf`:**
|
||||
|
||||
```conf
|
||||
bind 10.0.0.1
|
||||
|
@ -222,7 +222,7 @@ sentinel down-after-milliseconds gitlab-redis 10000
|
|||
sentinel failover_timeout 30000
|
||||
```
|
||||
|
||||
Sentinel 2 - `sentinel.conf`
|
||||
**Example configation for Sentinel 2 - `sentinel.conf`:**
|
||||
|
||||
```conf
|
||||
bind 10.0.0.2
|
||||
|
@ -233,7 +233,7 @@ sentinel down-after-milliseconds gitlab-redis 10000
|
|||
sentinel failover_timeout 30000
|
||||
```
|
||||
|
||||
Sentinel 3 - `sentinel.conf`
|
||||
**Example configation for Sentinel 3 - `sentinel.conf`:**
|
||||
|
||||
```conf
|
||||
bind 10.0.0.3
|
||||
|
|
Loading…
Reference in New Issue