2016-04-22 18:18:06 +00:00
# Configuring Redis for GitLab HA
2016-11-10 14:51:13 +00:00
High Availability with [Redis] is possible using a **Master** x **Slave**
topology with a [Redis Sentinel][sentinel] service to watch and automatically
start failover proceedings.
2016-11-10 06:07:30 +00:00
You can choose to install and manage Redis and Sentinel yourself, use
2016-11-10 14:51:13 +00:00
a hosted cloud solution or you can use the one that comes bundled with
Omnibus GitLab packages.
2016-04-22 18:18:06 +00:00
2016-11-10 14:51:13 +00:00
> **Notes:**
- Redis requires authentication for High Availability. See
2016-07-19 11:27:50 +00:00
[Redis Security ](http://redis.io/topics/security ) documentation for more
information. We recommend using a combination of a Redis password and tight
firewall rules to secure your Redis service.
2016-11-10 14:51:13 +00:00
- You are highly encouraged to read the [Redis Sentinel][sentinel] documentation
before configuring Redis HA with GitLab to fully understand the topology and
architecture.
2016-11-15 13:07:38 +00:00
- This is the documentation for the Omnibus GitLab packages. For installations
2016-11-15 15:08:53 +00:00
from source, follow the [Redis HA source installation ](redis_source.md ) guide.
2016-11-15 13:07:38 +00:00
- Redis Sentinel daemon is bundled with Omnibus GitLab Enterprise Edition only.
2016-11-15 15:08:53 +00:00
For configuring Sentinel with the Omnibus GitLab Community Edition and
installations from source, follow the
[Redis HA source installation ](redis_source.md ) guide.
2016-07-19 11:27:50 +00:00
2016-10-15 03:40:15 +00:00
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE - RUN doctoc TO UPDATE -->
**Table of Contents**
2016-11-10 14:51:13 +00:00
- [Overview ](#overview )
- [Prerequisites ](#prerequisites )
2016-11-10 18:00:06 +00:00
- [High Availability with Sentinel ](#high-availability-with-sentinel )
2016-11-10 14:51:13 +00:00
- [Recommended setup ](#recommended-setup )
2016-11-10 18:00:06 +00:00
- [Available configuration setups ](#available-configuration-setups )
- [Redis setup overview ](#redis-setup-overview )
- [Sentinel setup overview ](#sentinel-setup-overview )
2016-11-15 15:08:53 +00:00
- [Configuring Redis HA ](#configuring-redis-ha )
- [Step 1. Configuring the Master Redis instance ](#step-1-configuring-the-master-redis-instance )
- [Step 2. Configuring the Slave Redis instances ](#step-2-configuring-the-slave-redis-instances )
- [Step 3. Configuring the Redis Sentinel instances ](#step-3-configuring-the-redis-sentinel-instances )
- [Step 4. Configuring the GitLab application ](#step-4-configuring-the-gitlab-application )
2016-11-10 18:00:06 +00:00
- [Switching from an existing single-machine installation to Redis HA ](#switching-from-an-existing-single-machine-installation-to-redis-ha )
2016-11-15 15:08:53 +00:00
- [Example of a minimal configuration with 1 master, 2 slaves and 3 sentinels ](#example-of-a-minimal-configuration-with-1-master-2-slaves-and-3-sentinels )
2016-11-10 18:00:06 +00:00
- [Configuration for Redis master ](#configuration-for-redis-master )
- [Configuration for Redis slaves ](#configuration-for-redis-slaves )
- [Configuration for Sentinels ](#configuration-for-sentinels )
- [Advanced configuration ](#advanced-configuration )
2016-11-10 14:51:13 +00:00
- [Control running services ](#control-running-services )
2016-10-18 17:45:57 +00:00
- [Troubleshooting ](#troubleshooting )
2016-11-10 18:00:06 +00:00
- [Troubleshooting Redis replication ](#troubleshooting-redis-replication )
- [Troubleshooting Sentinel ](#troubleshooting-sentinel )
2016-11-10 12:21:34 +00:00
- [Changelog ](#changelog )
2016-11-15 15:08:53 +00:00
- [Experimental Redis Sentinel support ](#experimental-redis-sentinel-support )
2016-10-15 03:40:15 +00:00
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
2016-11-10 14:51:13 +00:00
## Overview
Before diving into the details of setting up Redis and Redis Sentinel for HA,
2016-11-10 18:00:06 +00:00
make sure you read this Overview section to better understand how the components
are tied together.
2016-11-10 14:51:13 +00:00
2016-11-10 18:00:06 +00:00
You need at least `3` independent machines: physical, or VMs running into
distinct physical machines. It is essential that all master and slaves Redis
instances run in different machines. If you fail to provision the machines in
that specific way, any issue with the shared environment can bring your entire
setup down.
2016-07-19 11:27:50 +00:00
2016-11-10 18:00:06 +00:00
It is OK to run a Sentinel along with a master or slave Redis instance.
No more than one Sentinel in the same machine though.
2016-07-19 11:27:50 +00:00
2016-11-10 18:00:06 +00:00
You also need to take in consideration the underlying network topology,
making sure you have redundant connectivity between Redis / Sentinel and
GitLab instances, otherwise the networks will become a single point of
failure.
2016-07-19 11:27:50 +00:00
2016-11-15 13:07:38 +00:00
Make sure that you read this document once as a whole before configuring the
components below.
2016-07-19 11:27:50 +00:00
2016-11-10 14:51:13 +00:00
### High Availability with Sentinel
2016-07-19 11:27:50 +00:00
2016-11-10 14:51:13 +00:00
>
2016-11-15 13:07:38 +00:00
- Starting with GitLab `8.11` , you can configure a list of Redis Sentinel
servers that will monitor a group of Redis servers to provide failover support.
- Starting with GitLab `8.14` , the Omnibus GitLab Enterprise Edition package
2016-11-15 15:08:53 +00:00
comes with Redis Sentinel daemon support.
2016-07-19 11:27:50 +00:00
2016-11-10 06:07:30 +00:00
High Availability with Redis requires a few things:
2016-07-19 11:27:50 +00:00
2016-11-10 06:07:30 +00:00
- Multiple Redis instances
2016-11-15 15:08:53 +00:00
- Run Redis in a **Master** x **Slave** topology
- Multiple Sentinel instances
2016-11-10 12:14:22 +00:00
- Application support and visibility to all Sentinel and Redis instances
2016-07-19 11:27:50 +00:00
2016-11-15 15:08:53 +00:00
Redis Sentinel can handle the most important tasks in an HA environment and that's
2016-11-10 18:00:06 +00:00
to help keep servers online with minimal to no downtime. Redis Sentinel:
2016-10-18 17:45:57 +00:00
2016-11-10 06:07:30 +00:00
- Monitors **Master** and **Slaves** instances to see if they are available
2016-11-10 14:51:13 +00:00
- Promotes a **Slave** to **Master** when the **Master** fails
2016-11-10 18:00:06 +00:00
- Demotes a **Master** to **Slave** when the failed **Master** comes back online
2016-11-10 14:51:13 +00:00
(to prevent data-partitioning)
2016-11-10 12:14:22 +00:00
- Can be queried by clients to always connect to the current **Master** server
2016-07-19 11:27:50 +00:00
2016-11-10 12:14:22 +00:00
When a **Master** fails to respond, it's the client's responsibility to handle
timeout and reconnect (querying a **Sentinel** for a new **Master** ).
2016-07-14 01:57:47 +00:00
To get a better understanding on how to correctly setup Sentinel, please read
2016-07-19 11:27:50 +00:00
the [Redis Sentinel documentation ](http://redis.io/topics/sentinel ) first, as
2016-11-10 18:00:06 +00:00
failing to configure it correctly can lead to data loss or can bring your
2016-11-10 06:07:30 +00:00
whole cluster down, invalidating the failover effort.
2016-07-19 11:27:50 +00:00
2016-11-10 14:51:13 +00:00
### Recommended setup
For a minimal setup, you will install the Omnibus GitLab package in `3`
2016-11-10 18:00:06 +00:00
**independent** machines, both with **Redis** and **Sentinel** :
2016-11-10 14:51:13 +00:00
- Redis Master + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
2016-11-10 18:00:06 +00:00
If you are not sure or don't understand why and where the amount of nodes come
2016-11-15 15:08:53 +00:00
from, read [Redis setup overview ](#redis-setup-overview ) and
[Sentinel setup overview ](#sentinel-setup-overview ).
2016-11-10 14:51:13 +00:00
2016-11-10 18:00:06 +00:00
For a recommended setup that can resist more failures, you will install
the Omnibus GitLab package in `5` **independent** machines, both with
2016-11-10 14:51:13 +00:00
**Redis** and **Sentinel** :
- Redis Master + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
2016-11-10 18:00:06 +00:00
### Available configuration setups
Based on your infrastructure setup and how you have installed GitLab, there are
multiple ways to configure Redis HA. Omnibus GitLab packages have Redis and/or
2016-11-15 13:07:38 +00:00
Redis Sentinel bundled with them so you only need to focus on configuration.
2016-11-10 18:00:06 +00:00
Pick the one that suits your needs.
- [Installations from source][source]: You need to install Redis and Sentinel
2016-11-15 13:07:38 +00:00
yourself. Use the [Redis HA installation from source ](redis_source.md ) guide.
- [Omnibus package Community Edition (CE) package][ce]: Redis is bundled, so you
can use the package with only the Redis service enabled (works for both master
and slave setups). To install and configure Sentinel, you can use the
[Redis HA installation from source ](redis_source.md ) guide.
- [Omnibus package Enterprise Edition (EE) package][ee]: Both Redis and Sentinel
are bundled, so you can use the EE package to setup the whole Redis HA
infrastructure (master, slave and Sentinel).
2016-11-15 15:08:53 +00:00
Note that even if you have installed GitLab using the Omnibus GitLab packages
(both CE and EE), you can still use an
[external Redis server ](#using-a-non-omnibus-external-redis-server ).
2016-11-10 14:51:13 +00:00
2016-11-10 18:00:06 +00:00
### Redis setup overview
2016-07-14 01:57:47 +00:00
2016-11-10 06:07:30 +00:00
You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they
need to be each in a independent machine (see explanation above).
You can have additional Redis nodes, that will help survive a situation
where more nodes goes down. Whenever there is only `2` nodes online, a failover
will not be initiated.
As an example, if you have `6` Redis nodes, a maximum of `3` can be
simultaneously down.
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
Please note that there are different requirements for Sentinel nodes.
If you host them in the same Redis machines, you may need to take
2016-11-10 12:14:22 +00:00
that restrictions into consideration when calculating the amount of
2016-11-10 06:07:30 +00:00
nodes to be provisioned. See [Sentinel setup ](#sentinel-setup )
documentation for more information.
All Redis nodes should be configured the same way and with similar server specs, as
in a failover situation, any **Slave** can be promoted as the new **Master** by
2016-07-19 11:27:50 +00:00
the Sentinel servers.
2016-07-14 01:57:47 +00:00
2016-11-10 06:07:30 +00:00
The replication requires authentication, so you need to define a password to
protect all Redis nodes and the Sentinels. They will all share the same
password, and all instances must be able to talk to
2016-10-15 03:40:15 +00:00
each other over the network.
2016-07-14 01:57:47 +00:00
2016-11-10 18:00:06 +00:00
### Sentinel setup overview
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
Sentinels watch both other Sentinels and Redis nodes. Whenever a Sentinel
2016-11-10 06:07:30 +00:00
detects that a Redis node is not responding, it will announce that to the
2016-11-15 15:08:53 +00:00
other Sentinels. They have to reach the **quorum** , that is the minimum amount
of Sentinels that agrees a node is down, in order to be able to start a failover.
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
Whenever the **quorum** is met, the **majority** of all known Sentinel nodes
need to be available and reachable, so that they can elect the Sentinel **leader**
2016-11-10 06:07:30 +00:00
who will take all the decisions to restore the service availability by:
- Promoting a new **Master**
- Reconfiguring the other **Slaves** and make them point to the new **Master**
- Announce the new **Master** to every other Sentinel peer
- Reconfigure the old **Master** and demote to **Slave** when it comes back online
You must have at least `3` Redis Sentinel servers, and they need to
2016-11-15 15:08:53 +00:00
be each in a independent machine (that are believed to fail independently),
ideally in different geographical areas.
2016-11-10 06:07:30 +00:00
You can configure them in the same machines where you've configured the other
Redis servers, but understand that if a whole node goes down, you loose both
a Sentinel and a Redis instance.
The number of sentinels should ideally always be an **odd** number, for the
consensus algorithm to be effective in the case of a failure.
In a `3` nodes topology, you can only afford `1` Sentinel node going down.
Whenever the **majority** of the Sentinels goes down, the network partition
protection prevents destructive actions and a failover **will not be started** .
Here are some examples:
- With `5` or `6` sentinels, a maximum of `2` can go down for a failover begin.
- With `7` sentinels, a maximum of `3` nodes can go down.
2016-11-15 15:08:53 +00:00
The **Leader** election can sometimes fail the voting round when **consensus**
2016-11-10 06:07:30 +00:00
is not achieved (see the odd number of nodes requirement above). In that case,
a new attempt will be made after the amount of time defined in
`sentinel['failover_timeout']` (in milliseconds).
2016-11-10 18:00:06 +00:00
>**Note:**
We will see where `sentinel['failover_timeout']` is defined later.
2016-11-15 15:08:53 +00:00
The `failover_timeout` variable has a lot of different use cases. According to
2016-11-10 18:00:06 +00:00
the official documentation:
2016-11-10 06:07:30 +00:00
- The time needed to re-start a failover after a previous failover was
already tried against the same master by a given Sentinel, is two
times the failover timeout.
- The time needed for a slave replicating to a wrong master according
to a Sentinel current configuration, to be forced to replicate
with the right master, is exactly the failover timeout (counting since
the moment a Sentinel detected the misconfiguration).
- The time needed to cancel a failover that is already in progress but
did not produced any configuration change (SLAVEOF NO ONE yet not
acknowledged by the promoted slave).
- The maximum time a failover in progress waits for all the slaves to be
reconfigured as slaves of the new master. However even after this time
the slaves will be reconfigured by the Sentinels anyway, but not with
the exact parallel-syncs progression as specified.
2016-11-15 15:08:53 +00:00
## Configuring Redis HA
2016-11-10 18:00:06 +00:00
This is the section where we install and setup the new Redis instances.
2016-07-14 01:57:47 +00:00
2016-11-10 18:00:06 +00:00
>**Notes:**
- We assume that you install GitLab and all HA components from scratch. If you
already have it installed and running, read how to
[switch from a single-machine installation to Redis HA ](#switching-from-an-existing-single-machine-installation-to-redis-ha ).
- Redis nodes (both master and slaves) will need the same password defined in
2016-11-15 13:07:38 +00:00
`redis['password']` . At any time during a failover the Sentinels can
reconfigure a node and change its status from master to slave and vice versa.
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
### Prerequisites
2016-11-10 14:51:13 +00:00
2016-11-15 15:08:53 +00:00
The prerequisites for a HA Redis setup are the following:
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
1. Provision the minimum required number of instances as specified in the
[recommended setup ](#recommended-setup ) section.
1. **Do NOT** install Redis or Redis Sentinel in the same machines your
GitLab application is running on. You can however opt in to install Redis
and Sentinel in the same machine (each in independent ones is recommended
though).
1. All Redis nodes must be able to talk to each other and accept incoming
connections over Redis (`6379`) and Sentinel (`26379`) ports (unless you
change the default ones).
1. The server that hosts the GitLab application must be able to access the
Redis nodes.
1. Protect the nodes from access from external networks (Internet), using
firewall.
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
### Step 1. Configuring the master Redis instance
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
1. SSH into the **master** Redis server and login as root:
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
```
sudo -i
```
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
1. [Download/install ](https://about.gitlab.com/installation ) the Omnibus GitLab
package using **steps 1 and 2** from the GitLab downloads page.
- Make sure you select the correct Omnibus package, with the same version
and type (Community, Enterprise editions) of your current install.
- Do not complete any other steps on the download page.
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
```ruby
# Enable the master role and disable all other services in the machine
# (you can still enable Sentinel).
redis_master_role['enable'] = true
# IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces.
# If you really need to bind to an external accessible IP, make
# sure you add extra firewall rules to prevent unauthorized access.
redis['bind'] = '10.0.0.1'
# Define a port so Redis can listen for TCP requests which will allow other
# machines to connect to it.
redis['port'] = 6379
# Set up password authentication for Redis (use the same password in all nodes).
redis['password'] = 'redis-password-goes-here'
```
1. To prevent database migrations from running on upgrade, run:
```
touch /etc/gitlab/skip-auto-migrations
```
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
Only the primary GitLab application server should handle migrations.
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
2016-07-19 11:27:50 +00:00
2016-11-15 15:08:53 +00:00
### Step 2. Configuring the slave Redis instances
2016-10-15 03:40:15 +00:00
2016-11-15 15:08:53 +00:00
1. SSH into the **slave** Redis server and login as root:
2016-10-15 03:40:15 +00:00
2016-11-15 15:08:53 +00:00
```
sudo -i
```
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
1. [Download/install ](https://about.gitlab.com/installation ) the Omnibus GitLab
package using **steps 1 and 2** from the GitLab downloads page.
- Make sure you select the correct Omnibus package, with the same version
and type (Community, Enterprise editions) of your current install.
- Do not complete any other steps on the download page.
2016-10-15 03:40:15 +00:00
2016-11-15 15:08:53 +00:00
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby
# Enable the slave role and disable all other services in the machine
# (you can still enable Sentinel). This will also set automatically
# `redis['master'] = false` .
redis_slave_role['enable'] = true
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
# IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces.
# If you really need to bind to an external accessible IP, make
# sure you add extra firewall rules to prevent unauthorized access.
redis['bind'] = '10.0.0.2'
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
# Define a port so Redis can listen for TCP requests which will allow other
# machines to connect to it.
redis['port'] = 6379
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
# The same password for Redeis authentication you set up for the master node.
redis['password'] = 'redis-password-goes-here'
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
# The IP of the master Redis node.
redis['master_ip'] = '10.0.0.1'
2016-11-10 18:00:06 +00:00
2016-11-15 15:08:53 +00:00
# Port of master Redis server, uncomment to change to non default. Defaults
# to `6379` .
#redis ['master_port'] = 6379
```
1. To prevent database migrations from running on upgrade, run:
```
touch /etc/gitlab/skip-auto-migrations
```
Only the primary GitLab application server should handle migrations.
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
1. Go through the steps again for all the other slave nodes.
---
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
2016-11-10 18:00:06 +00:00
a failover, as the nodes will be managed by the Sentinels, and even after a
`gitlab-ctl reconfigure` , they will get their configuration restored by
the same Sentinels.
2016-11-15 15:08:53 +00:00
### Step 3. Configuring the Redis Sentinel instances
2016-11-10 18:00:06 +00:00
>**Note:**
2016-11-15 15:08:53 +00:00
Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. The
following section assumes you are using Omnibus GitLab Enterprise Edition.
For the Omnibus Community Edition and installations from source, follow the
[Redis HA source install ](redis_source.md ) guide.
2016-11-10 06:07:30 +00:00
Now that the Redis servers are all set up, let's configure the Sentinel
servers.
If you are not sure if your Redis servers are working and replicating
correctly, please read the [Troubleshooting Replication ](#troubleshooting-replication )
and fix it before proceeding with Sentinel setup.
You must have at least `3` Redis Sentinel servers, and they need to
2016-11-15 15:08:53 +00:00
be each in an independent machine. You can configure them in the same
2016-11-10 06:07:30 +00:00
machines where you've configured the other Redis servers.
2016-11-15 15:08:53 +00:00
With GitLab Enterprise Edition, you can use the Omnibus package to setup
multiple machines with the Sentinel daemon.
---
1. SSH into the server that will host Redis Sentinel and login as root:
```
sudo -i
```
1. **You can omit this step if the Sentinels will be hosted in the same node as
the other Redis instances.**
[Download/install ](https://about.gitlab.com/downloads-ee ) the
Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the
GitLab downloads page.
- Make sure you select the correct Omnibus package, with the same version
the GitLab application is running.
- Do not complete any other steps on the download page.
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
1. Edit `/etc/gitlab/gitlab.rb` and add the contents (if you are installing the
Sentinels in the same node as the other Redis instances, some values might
be duplicate below):
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
```ruby
redis_sentinel_role['enable'] = true
# Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis'
# The same password for Redis authentication you set up for the master node.
redis['password'] = 'redis-password-goes-here'
# The IP of the master Redis node.
redis['master_ip'] = '10.0.0.1'
# Define a port so Redis can listen for TCP requests which will allow other
# machines to connect to it.
redis['port'] = 6379
# Port of master Redis server, uncomment to change to non default. Defaults
# to `6379` .
#redis ['master_port'] = 6379
## Configure Sentinel
sentinel['bind'] = '10.0.0.1'
# Port that Sentinel listens on, uncomment to change to non default. Defaults
# to `26379` .
# sentinel['port'] = 26379
## Quorum must reflect the amount of voting sentinels it take to start a failover.
## Value must NOT be greater then the amount of sentinels.
##
## The quorum can be used to tune Sentinel in two ways:
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
## we deploy, we are basically making Sentinel more sensible to master failures,
## triggering a failover as soon as even just a minority of Sentinels is no longer
## able to talk with the master.
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
## making Sentinel able to failover only when there are a very large number (larger
## than majority) of well connected Sentinels which agree about the master being down.s
sentinel['quorum'] = 2
## Consider unresponsive server down after x amount of ms.
# sentinel['down_after_milliseconds'] = 10000
## Specifies the failover timeout in milliseconds. It is used in many ways:
##
## - The time needed to re-start a failover after a previous failover was
## already tried against the same master by a given Sentinel, is two
## times the failover timeout.
##
## - The time needed for a slave replicating to a wrong master according
## to a Sentinel current configuration, to be forced to replicate
## with the right master, is exactly the failover timeout (counting since
## the moment a Sentinel detected the misconfiguration).
##
## - The time needed to cancel a failover that is already in progress but
## did not produced any configuration change (SLAVEOF NO ONE yet not
## acknowledged by the promoted slave).
##
## - The maximum time a failover in progress waits for all the slaves to be
## reconfigured as slaves of the new master. However even after this time
## the slaves will be reconfigured by the Sentinels anyway, but not with
## the exact parallel-syncs progression as specified.
# sentinel['failover_timeout'] = 60000
```
1. To prevent database migrations from running on upgrade, run:
```
touch /etc/gitlab/skip-auto-migrations
```
Only the primary GitLab application server should handle migrations.
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
1. Go through the steps again for all the other Sentinel nodes.
### Step 4. Configuring the GitLab application
2016-11-10 06:07:30 +00:00
The final part is to inform the main GitLab application server of the Redis
Sentinels servers and authentication credentials.
You can enable or disable Sentinel support at any time in new or existing
installations. From the GitLab application perspective, all it requires is
the correct credentials for the Sentinel nodes.
While it doesn't require a list of all Sentinel nodes, in case of a failure,
2016-11-10 12:14:22 +00:00
it needs to access at least one of the listed.
2016-11-10 06:07:30 +00:00
>**Note:**
The following steps should be performed in the [GitLab application server ](gitlab.md )
2016-11-10 18:00:06 +00:00
which ideally should not have Redis or Sentinels on it for a HA setup.
2016-11-10 06:07:30 +00:00
1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:
2016-11-15 15:08:53 +00:00
```
# Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis'
# The same password for Redis authentication you set up for the master node.
redis['password'] = 'redis-password-goes-here'
# A list of sentinels with `host` and `port`
gitlab_rails['redis_sentinels'] = [
{'host' => '10.0.0.1', 'port' => 26379},
{'host' => '10.0.0.2', 'port' => 26379},
{'host' => '10.0.0.3', 'port' => 26379}
]
```
2016-11-10 06:07:30 +00:00
2016-11-10 18:00:06 +00:00
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
2016-11-10 06:07:30 +00:00
2016-11-10 18:00:06 +00:00
## Switching from an existing single-machine installation to Redis HA
If you already have a single-machine GitLab install running, you will need to
replicate from this machine first, before de-activating the Redis instance
inside it.
Your single-machine install will be the initial **Master** , and the `3` others
should be configured as **Slave** pointing to this machine.
After replication catches up, you will need to stop services in the
single-machine install, to rotate the **Master** to one of the new nodes.
Make the required changes in configuration and restart the new nodes again.
To disable redis in the single install, edit `/etc/gitlab/gitlab.rb` :
```ruby
redis['enable'] = false
```
If you fail to replicate first, you may loose data (unprocessed background jobs).
2016-11-15 13:07:38 +00:00
## Example of a minimal configuration with 1 master, 2 slaves and 3 sentinels
2016-11-10 06:07:30 +00:00
In this example we consider that all servers have an internal network
interface with IPs in the `10.0.0.x` range, and that they can connect
to each other using these IPs.
In a real world usage, you would also setup firewall rules to prevent
2016-11-10 18:00:06 +00:00
unauthorized access from other machines and block traffic from the
2016-11-10 06:07:30 +00:00
outside (Internet).
We will use the same `3` nodes with **Redis** + **Sentinel** topology
2016-11-15 15:08:53 +00:00
discussed in [Redis setup overview ](#redis-setup-overview ) and
[Sentinel setup overview ](#sentinel-setup-overview ) documentation.
2016-11-10 06:07:30 +00:00
Here is a list and description of each **machine** and the assigned **IP** :
* `10.0.0.1` : Redis Master + Sentinel 1
* `10.0.0.2` : Redis Slave 1 + Sentinel 2
2016-11-15 15:08:53 +00:00
* `10.0.0.3` : Redis Slave 2 + Sentinel 3
* `10.0.0.4` : GitLab application
2016-11-10 06:07:30 +00:00
Please note that after the initial configuration, if a failover is initiated
by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master**
will change permanently (including in `redis.conf` ) from one node to the other,
until a new failover is initiated again.
2016-11-10 12:14:22 +00:00
The same thing will happen with `sentinel.conf` that will be overridden after the
2016-11-10 06:07:30 +00:00
initial execution, after any new sentinel node starts watching the **Master** ,
or a failover promotes a different **Master** node.
2016-11-15 15:08:53 +00:00
### Example configuration for Redis master
2016-11-10 06:07:30 +00:00
**Example configation for Redis Master:**
2016-07-19 11:27:50 +00:00
2016-10-18 17:45:57 +00:00
In `/etc/gitlab/gitlab.rb` :
2016-10-15 03:40:15 +00:00
```ruby
2016-11-10 06:07:30 +00:00
redis_master_role['enable'] = true
redis['bind'] = '10.0.0.1'
2016-10-15 03:40:15 +00:00
redis['port'] = 6379
2016-10-28 22:40:28 +00:00
redis['password'] = 'redis-password-goes-here'
2016-10-15 03:40:15 +00:00
```
2016-11-10 18:00:06 +00:00
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
2016-11-15 15:08:53 +00:00
### Example configuration for Redis slaves
2016-11-10 06:07:30 +00:00
2016-11-15 15:08:53 +00:00
**Example configuration for Slave 1:**
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
In `/etc/gitlab/gitlab.rb` :
2016-10-15 03:40:15 +00:00
```ruby
2016-11-10 06:07:30 +00:00
redis_slave_role['enable'] = true
redis['bind'] = '10.0.0.2'
2016-10-15 03:40:15 +00:00
redis['port'] = 6379
2016-10-28 22:40:28 +00:00
redis['password'] = 'redis-password-goes-here'
2016-10-15 03:40:15 +00:00
redis['master_password'] = 'redis-password-goes-here'
2016-11-10 06:07:30 +00:00
redis['master_ip'] = '10.0.0.1' # IP of master Redis server
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
2016-10-15 03:40:15 +00:00
```
2016-11-15 15:08:53 +00:00
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
2016-07-19 11:27:50 +00:00
2016-11-15 15:08:53 +00:00
**Example configuration for Slave 2:**
2016-07-19 11:27:50 +00:00
2016-11-10 06:07:30 +00:00
In `/etc/gitlab/gitlab.rb` :
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
```ruby
redis_slave_role['enable'] = true
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
redis['bind'] = '10.0.0.3'
redis['port'] = 6379
redis['password'] = 'redis-password-goes-here'
redis['master_password'] = 'redis-password-goes-here'
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
redis['master_ip'] = '10.0.0.1' # IP of master Redis server
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
```
2016-10-18 17:45:57 +00:00
2016-11-10 18:00:06 +00:00
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
2016-11-15 15:08:53 +00:00
### Example configuration for Sentinels
2016-07-14 01:57:47 +00:00
2016-11-10 18:00:06 +00:00
>**Note:**
Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the
Omnibus Community Edition and installations from source, follow the
[Redis HA source install ](redis_source.md ) guide.
2016-08-02 01:25:28 +00:00
2016-11-10 06:07:30 +00:00
Please note that some of the variables are already configured previously
as they are required for Redis replication.
2016-07-14 01:57:47 +00:00
2016-11-15 15:08:53 +00:00
**Example configuration for Sentinel 1:**
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
In `/etc/gitlab/gitlab.rb` :
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
```ruby
redis_sentinel_role['enable'] = true
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
2016-11-10 18:00:06 +00:00
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
2016-11-10 06:07:30 +00:00
redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
## Configure Sentinel
sentinel['bind'] = '10.0.0.1'
# sentinel['port'] = 26379 # uncomment to change default port
sentinel['quorum'] = 2
2016-11-08 05:24:43 +00:00
2016-11-10 06:07:30 +00:00
## Consider unresponsive server down after x amount of ms.
# sentinel['down_after_milliseconds'] = 10000
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
# sentinel['failover_timeout'] = 60000
```
2016-07-14 01:57:47 +00:00
2016-11-15 15:08:53 +00:00
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
**Example configuration for Sentinel 2:**
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
In `/etc/gitlab/gitlab.rb` :
2016-10-15 03:40:15 +00:00
```ruby
2016-11-10 06:07:30 +00:00
redis_sentinel_role['enable'] = true
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
2016-11-10 18:00:06 +00:00
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
2016-11-10 06:07:30 +00:00
redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
## Configure Sentinel
sentinel['bind'] = '10.0.0.2'
# sentinel['port'] = 26379 # uncomment to change default port
2016-10-15 03:40:15 +00:00
2016-11-10 06:07:30 +00:00
sentinel['quorum'] = 2
## Consider unresponsive server down after x amount of ms.
# sentinel['down_after_milliseconds'] = 10000
# sentinel['failover_timeout'] = 60000
```
2016-10-15 03:40:15 +00:00
2016-11-15 15:08:53 +00:00
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
**Example configuration for Sentinel 3:**
2016-11-10 06:07:30 +00:00
In `/etc/gitlab/gitlab.rb` :
```ruby
2016-10-28 22:40:28 +00:00
redis_sentinel_role['enable'] = true
2016-10-15 03:40:15 +00:00
2016-10-18 17:45:57 +00:00
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
2016-11-10 18:00:06 +00:00
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
2016-10-18 17:45:57 +00:00
redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance
2016-11-10 06:07:30 +00:00
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
2016-10-18 17:45:57 +00:00
## Configure Sentinel
2016-11-10 06:07:30 +00:00
sentinel['bind'] = '10.0.0.3'
2016-10-18 17:45:57 +00:00
# sentinel['port'] = 26379 # uncomment to change default port
sentinel['quorum'] = 2
## Consider unresponsive server down after x amount of ms.
# sentinel['down_after_milliseconds'] = 10000
# sentinel['failover_timeout'] = 60000
```
2016-10-15 03:40:15 +00:00
2016-11-15 15:08:53 +00:00
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
### Example configuration for the GitLab application
In `/etc/gitlab/gitlab.rb` :
```
# Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis'
# The same password for Redis authentication you set up for the master node.
redis['password'] = 'redis-password-goes-here'
# A list of sentinels with `host` and `port`
gitlab_rails['redis_sentinels'] = [
{'host' => '10.0.0.1', 'port' => 26379},
{'host' => '10.0.0.2', 'port' => 26379},
{'host' => '10.0.0.3', 'port' => 26379}
]
```
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
2016-11-10 18:00:06 +00:00
## Advanced configuration
Omnibus GitLab configures some things behind the curtains to make the sysadmins'
lives easier. If you want to know what happens underneath keep reading.
2016-11-10 06:07:30 +00:00
### Control running services
2016-11-15 15:08:53 +00:00
In the previous example, we've used `redis_sentinel_role` and
`redis_master_role` which simplifies the amount of configuration changes.
2016-10-28 22:40:28 +00:00
If you want more control, here is what each one sets for you automatically
when enabled:
```ruby
## Redis Sentinel Role
redis_sentinel_role['enable'] = true
2016-11-15 15:08:53 +00:00
# When Sentinel Role is enabled, the following services are also enabled
2016-10-28 22:40:28 +00:00
sentinel['enable'] = true
2016-11-15 15:08:53 +00:00
# The following services are disabled
2016-10-28 22:40:28 +00:00
redis['enable'] = false
bootstrap['enable'] = false
nginx['enable'] = false
postgresql['enable'] = false
gitlab_rails['enable'] = false
mailroom['enable'] = false
2016-11-15 15:08:53 +00:00
-------
## Redis master/slave Role
2016-10-28 22:40:28 +00:00
redis_master_role['enable'] = true # enable only one of them
redis_slave_role['enable'] = true # enable only one of them
2016-11-15 15:08:53 +00:00
# When Redis Master or Slave role are enabled, the following services are
# enabled/disabled. Note that if Redis and Sentinel roles are combined, both
# services will be enabled.
2016-10-28 22:40:28 +00:00
2016-11-15 15:08:53 +00:00
# The following services are disabled
2016-10-28 22:40:28 +00:00
sentinel['enable'] = false
bootstrap['enable'] = false
nginx['enable'] = false
postgresql['enable'] = false
gitlab_rails['enable'] = false
mailroom['enable'] = false
2016-11-15 15:08:53 +00:00
# For Redis Slave role, also change this setting from default 'true' to 'false':
2016-10-28 22:40:28 +00:00
redis['master'] = false
```
2016-07-19 11:27:50 +00:00
2016-11-10 18:00:06 +00:00
You can find the relevant attributes defined in [gitlab_rails.rb][omnifile].
2016-10-18 17:45:57 +00:00
## Troubleshooting
There are a lot of moving parts that needs to be taken care carefully
in order for the HA setup to work as expected.
2016-11-10 18:00:06 +00:00
Before proceeding with the troubleshooting below, check your firewall rules:
2016-10-18 17:45:57 +00:00
- Redis machines
- Accept TCP connection in `6379`
- Connect to the other Redis machines via TCP in `6379`
- Sentinel machines
- Accept TCP connection in `26379`
- Connect to other Sentinel machines via TCP in `26379`
- Connect to the Redis machines via TCP in `6379`
2016-11-10 18:00:06 +00:00
### Troubleshooting Redis replication
2016-10-18 17:45:57 +00:00
You can check if everything is correct by connecting to each server using
`redis-cli` application, and sending the `INFO` command.
If authentication was correctly defined, it should fail with:
`NOAUTH Authentication required` error. Try to authenticate with the
previous defined password with `AUTH redis-password-goes-here` and
try the `INFO` command again.
Look for the `# Replication` section where you should see some important
information like the `role` of the server.
When connected to a `master` redis, you will see the number of connected
`slaves` , and a list of each with connection details:
```
# Replication
role:master
connected_slaves:1
slave0:ip=10.133.5.21,port=6379,state=online,offset=208037514,lag=1
master_repl_offset:208037658
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:206989083
repl_backlog_histlen:1048576
```
When it's a `slave` , you will see details of the master connection and if
its `up` or `down` :
```
# Replication
role:slave
master_host:10.133.1.58
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:208096498
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
```
2016-11-10 18:00:06 +00:00
### Troubleshooting Sentinel
2016-10-06 23:47:22 +00:00
If you get an error like: `Redis::CannotConnectError: No sentinels available.` ,
there may be something wrong with your configuration files or it can be related
to [this issue][gh-531].
You must make sure you are defining the same value in `redis['master_name']`
and `redis['master_pasword']` as you defined for your sentinel node.
The way the redis connector `redis-rb` works with sentinel is a bit
non-intuitive. We try to hide the complexity in omnibus, but it still requires
a few extra configs.
2016-07-19 11:27:50 +00:00
---
2016-04-22 18:18:06 +00:00
2016-07-19 11:27:50 +00:00
To make sure your configuration is correct:
2016-04-22 18:18:06 +00:00
2016-07-19 11:27:50 +00:00
1. SSH into your GitLab application server
1. Enter the Rails console:
```
# For Omnibus installations
sudo gitlab-rails console
# For source installations
2016-10-15 03:40:15 +00:00
sudo -u git rails console production
2016-07-19 11:27:50 +00:00
```
1. Run in the console:
2016-04-22 18:18:06 +00:00
```ruby
2016-07-19 11:27:50 +00:00
redis = Redis.new(Gitlab::Redis.params)
redis.info
2016-04-22 18:18:06 +00:00
```
2016-07-19 11:27:50 +00:00
Keep this screen open and try to simulate a failover below.
2016-04-22 18:18:06 +00:00
2016-07-19 11:27:50 +00:00
1. To simulate a failover on master Redis, SSH into the Redis server and run:
```bash
2016-11-08 05:24:43 +00:00
# port must match your master redis port, and the sleep time must be a few seconds bigger than defined one
redis-cli -h localhost -p 6379 DEBUG sleep 20
2016-07-19 11:27:50 +00:00
```
1. Then back in the Rails console from the first step, run:
```
redis.info
```
You should see a different port after a few seconds delay
(the failover/reconnect time).
2016-04-22 18:18:06 +00:00
2016-11-10 12:21:34 +00:00
## Changelog
Changes to Redis HA over time.
2016-11-15 15:08:53 +00:00
**8.14**
- Redis Sentinel support is production-ready and bundled in the Omnibus GitLab
Enterprise Edition package
- Documentation restructure for better readability
**8.11**
- Experimental Redis Sentinel support was added
2016-11-10 12:21:34 +00:00
### Experimental Redis Sentinel support
>
Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11.
Starting with 8.14, Redis Sentinel is no longer experimental.
If you used with versions `< 8.14` before, please check the updated
documentation here.
2016-04-22 18:18:06 +00:00
---
Read more on high-availability configuration:
1. [Configure the database ](database.md )
1. [Configure NFS ](nfs.md )
1. [Configure the GitLab application servers ](gitlab.md )
1. [Configure the load balancers ](load_balancer.md )
2016-07-19 11:27:50 +00:00
[ce-1877]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1877
[restart]: ../restart_gitlab.md#installations-from-source
[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure
[gh-531]: https://github.com/redis/redis-rb/issues/531
[gh-534]: https://github.com/redis/redis-rb/issues/534
2016-11-10 14:51:13 +00:00
[redis]: http://redis.io/
[sentinel]: http://redis.io/topics/sentinel
2016-11-15 15:08:53 +00:00
[omnifile]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb
2016-11-10 18:00:06 +00:00
[source]: ../../install/installation.md
[ce]: https://about.gitlab.com/downloads
[ee]: https://about.gitlab.com/downloads-ee