2016-04-22 18:18:06 +00:00
|
|
|
# Configuring Redis for GitLab HA
|
|
|
|
|
2016-11-10 14:51:13 +00:00
|
|
|
High Availability with [Redis] is possible using a **Master** x **Slave**
|
|
|
|
topology with a [Redis Sentinel][sentinel] service to watch and automatically
|
|
|
|
start failover proceedings.
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
You can choose to install and manage Redis and Sentinel yourself, use
|
2016-11-10 14:51:13 +00:00
|
|
|
a hosted cloud solution or you can use the one that comes bundled with
|
|
|
|
Omnibus GitLab packages.
|
2016-04-22 18:18:06 +00:00
|
|
|
|
2016-11-10 14:51:13 +00:00
|
|
|
> **Notes:**
|
|
|
|
- Redis requires authentication for High Availability. See
|
2016-07-19 11:27:50 +00:00
|
|
|
[Redis Security](http://redis.io/topics/security) documentation for more
|
|
|
|
information. We recommend using a combination of a Redis password and tight
|
|
|
|
firewall rules to secure your Redis service.
|
2016-11-10 14:51:13 +00:00
|
|
|
- You are highly encouraged to read the [Redis Sentinel][sentinel] documentation
|
|
|
|
before configuring Redis HA with GitLab to fully understand the topology and
|
|
|
|
architecture.
|
2016-11-10 18:00:06 +00:00
|
|
|
- This is the documentation for the Omnibus packages. For installations from
|
|
|
|
source, follow the [Redis HA source install](redis_source.md) guide.
|
|
|
|
- Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the
|
|
|
|
Omnibus Community Edition and installations from source, follow the
|
|
|
|
[Redis HA source install](redis_source.md) guide.
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-10-15 03:40:15 +00:00
|
|
|
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
|
|
|
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
|
|
|
**Table of Contents**
|
|
|
|
|
2016-11-10 14:51:13 +00:00
|
|
|
- [Overview](#overview)
|
|
|
|
- [Prerequisites](#prerequisites)
|
2016-11-10 18:00:06 +00:00
|
|
|
- [High Availability with Sentinel](#high-availability-with-sentinel)
|
2016-11-10 14:51:13 +00:00
|
|
|
- [Recommended setup](#recommended-setup)
|
2016-11-10 18:00:06 +00:00
|
|
|
- [Available configuration setups](#available-configuration-setups)
|
|
|
|
- [Using a non-Omnibus external Redis server](#using-a-non-omnibus-external-redis-server)
|
|
|
|
- [Redis setup overview](#redis-setup-overview)
|
|
|
|
- [Sentinel setup overview](#sentinel-setup-overview)
|
2016-11-10 14:51:13 +00:00
|
|
|
- [Redis HA configuration](#redis-ha-configuration)
|
2016-11-10 18:00:06 +00:00
|
|
|
- [Configuring the Master Redis instance](#configuring-the-master-redis-instance)
|
|
|
|
- [Configuring the Slave Redis instances](#configuring-the-slave-redis-instances)
|
|
|
|
- [Configuring the Sentinel instances](#configuring-the-sentinel-instances)
|
|
|
|
- [Configuring the GitLab application](#configuring-the-gitlab-application)
|
|
|
|
- [Switching from an existing single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha)
|
2016-11-10 14:51:13 +00:00
|
|
|
- [Minimal example configuration with 1 master, 2 slaves and 3 sentinels](#minimal-example-configuration-with-1-master-2-slaves-and-3-sentinels)
|
2016-11-10 18:00:06 +00:00
|
|
|
- [Configuration for Redis master](#configuration-for-redis-master)
|
|
|
|
- [Configuration for Redis slaves](#configuration-for-redis-slaves)
|
|
|
|
- [Configuration for Sentinels](#configuration-for-sentinels)
|
|
|
|
- [Advanced configuration](#advanced-configuration)
|
2016-11-10 14:51:13 +00:00
|
|
|
- [Control running services](#control-running-services)
|
2016-10-18 17:45:57 +00:00
|
|
|
- [Troubleshooting](#troubleshooting)
|
2016-11-10 18:00:06 +00:00
|
|
|
- [Troubleshooting Redis replication](#troubleshooting-redis-replication)
|
|
|
|
- [Troubleshooting Sentinel](#troubleshooting-sentinel)
|
2016-11-10 12:21:34 +00:00
|
|
|
- [Changelog](#changelog)
|
2016-11-10 14:51:13 +00:00
|
|
|
- [Experimental Redis Sentinel support](#experimental-redis-sentinel-support)
|
2016-10-15 03:40:15 +00:00
|
|
|
|
|
|
|
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
|
|
|
|
2016-11-10 14:51:13 +00:00
|
|
|
## Overview
|
|
|
|
|
|
|
|
Before diving into the details of setting up Redis and Redis Sentinel for HA,
|
2016-11-10 18:00:06 +00:00
|
|
|
make sure you read this Overview section to better understand how the components
|
|
|
|
are tied together.
|
2016-11-10 14:51:13 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
### Prerequisites
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
You need at least `3` independent machines: physical, or VMs running into
|
|
|
|
distinct physical machines. It is essential that all master and slaves Redis
|
|
|
|
instances run in different machines. If you fail to provision the machines in
|
|
|
|
that specific way, any issue with the shared environment can bring your entire
|
|
|
|
setup down.
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
It is OK to run a Sentinel along with a master or slave Redis instance.
|
|
|
|
No more than one Sentinel in the same machine though.
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
You also need to take in consideration the underlying network topology,
|
|
|
|
making sure you have redundant connectivity between Redis / Sentinel and
|
|
|
|
GitLab instances, otherwise the networks will become a single point of
|
|
|
|
failure.
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
Read carefully how to configure the components below.
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 14:51:13 +00:00
|
|
|
### High Availability with Sentinel
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 14:51:13 +00:00
|
|
|
>
|
|
|
|
- Since GitLab `8.11`, you can configure a list of Redis Sentinel servers that
|
|
|
|
will monitor a group of Redis servers to provide failover support.
|
|
|
|
- With GitLab `8.14`, we bundled Redis Sentinel as part of Omnibus package and
|
|
|
|
improved the way you use and configure it.
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
High Availability with Redis requires a few things:
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
- Multiple Redis instances
|
|
|
|
- Run Redis in a **Master** x **Slave** topology
|
|
|
|
- Multiple Sentinel instances
|
2016-11-10 12:14:22 +00:00
|
|
|
- Application support and visibility to all Sentinel and Redis instances
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
Redis Sentinel can handle the most important tasks in a HA environment and that's
|
|
|
|
to help keep servers online with minimal to no downtime. Redis Sentinel:
|
2016-10-18 17:45:57 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
- Monitors **Master** and **Slaves** instances to see if they are available
|
2016-11-10 14:51:13 +00:00
|
|
|
- Promotes a **Slave** to **Master** when the **Master** fails
|
2016-11-10 18:00:06 +00:00
|
|
|
- Demotes a **Master** to **Slave** when the failed **Master** comes back online
|
2016-11-10 14:51:13 +00:00
|
|
|
(to prevent data-partitioning)
|
2016-11-10 12:14:22 +00:00
|
|
|
- Can be queried by clients to always connect to the current **Master** server
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 12:14:22 +00:00
|
|
|
When a **Master** fails to respond, it's the client's responsibility to handle
|
|
|
|
timeout and reconnect (querying a **Sentinel** for a new **Master**).
|
2016-07-14 01:57:47 +00:00
|
|
|
|
|
|
|
To get a better understanding on how to correctly setup Sentinel, please read
|
2016-07-19 11:27:50 +00:00
|
|
|
the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
|
2016-11-10 18:00:06 +00:00
|
|
|
failing to configure it correctly can lead to data loss or can bring your
|
2016-11-10 06:07:30 +00:00
|
|
|
whole cluster down, invalidating the failover effort.
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 14:51:13 +00:00
|
|
|
### Recommended setup
|
|
|
|
|
|
|
|
For a minimal setup, you will install the Omnibus GitLab package in `3`
|
2016-11-10 18:00:06 +00:00
|
|
|
**independent** machines, both with **Redis** and **Sentinel**:
|
2016-11-10 14:51:13 +00:00
|
|
|
|
|
|
|
- Redis Master + Sentinel
|
|
|
|
- Redis Slave + Sentinel
|
|
|
|
- Redis Slave + Sentinel
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
If you are not sure or don't understand why and where the amount of nodes come
|
|
|
|
from, read [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup).
|
2016-11-10 14:51:13 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
For a recommended setup that can resist more failures, you will install
|
|
|
|
the Omnibus GitLab package in `5` **independent** machines, both with
|
2016-11-10 14:51:13 +00:00
|
|
|
**Redis** and **Sentinel**:
|
|
|
|
|
|
|
|
- Redis Master + Sentinel
|
|
|
|
- Redis Slave + Sentinel
|
|
|
|
- Redis Slave + Sentinel
|
|
|
|
- Redis Slave + Sentinel
|
|
|
|
- Redis Slave + Sentinel
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
### Available configuration setups
|
|
|
|
|
|
|
|
Based on your infrastructure setup and how you have installed GitLab, there are
|
|
|
|
multiple ways to configure Redis HA. Omnibus GitLab packages have Redis and/or
|
|
|
|
Redis Sentinel bundled with them to save you the hassle to install them yourself.
|
|
|
|
Pick the one that suits your needs.
|
|
|
|
|
|
|
|
- [Installations from source][source]: You need to install Redis and Sentinel
|
|
|
|
yourself. Use the [Redis HA source install](redis_source.md) guide.
|
|
|
|
- [Omnibus package Community Edition (CE)][ce]: Redis is bundled, so you can use the
|
|
|
|
package with only the Redis service enabled (works for both master and slave
|
|
|
|
setups).
|
|
|
|
- [Omnibus package Enterprise Edition (EE)][ee]: Both Redis and Sentinel are bundled,
|
|
|
|
so you can use the EE package to setup the whole Redis HA infrastructure
|
|
|
|
(master, slave and Sentinel).
|
|
|
|
|
|
|
|
Note that if you have installed GitLab using the Omnibus packages (both CE and EE),
|
|
|
|
you can also use an [external Redis server](#using-a-non-omnibus-external-redis-server).
|
|
|
|
|
|
|
|
### Using a non-Omnibus external Redis server
|
|
|
|
|
|
|
|
If you're hosting GitLab on a cloud provider, you can optionally use a
|
|
|
|
managed service for Redis. For example, AWS offers a managed ElastiCache service
|
|
|
|
that runs Redis.
|
|
|
|
|
|
|
|
Managed services can provide High Availability using their own proprietary
|
|
|
|
technology and provide a transparent proxy (which means that GitLab doesn't
|
|
|
|
need any additional change) or they will use Sentinel and manage it for you.
|
|
|
|
|
|
|
|
If your provider uses Sentinel, see [GitLab Setup](#gitlab-setup)
|
|
|
|
to understand where you need to provide the list of servers and credentials.
|
|
|
|
|
|
|
|
If you want to setup Redis by yourself, without using Omnibus, you can
|
|
|
|
read the documentation on [configuring Redis HA for source installs](redis_source.md).
|
2016-11-10 14:51:13 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
### Redis setup overview
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they
|
|
|
|
need to be each in a independent machine (see explanation above).
|
|
|
|
|
|
|
|
You can have additional Redis nodes, that will help survive a situation
|
|
|
|
where more nodes goes down. Whenever there is only `2` nodes online, a failover
|
|
|
|
will not be initiated.
|
|
|
|
|
|
|
|
As an example, if you have `6` Redis nodes, a maximum of `3` can be
|
|
|
|
simultaneously down.
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
Please note that there are different requirements for Sentinel nodes.
|
|
|
|
If you host them in the same Redis machines, you may need to take
|
2016-11-10 12:14:22 +00:00
|
|
|
that restrictions into consideration when calculating the amount of
|
2016-11-10 06:07:30 +00:00
|
|
|
nodes to be provisioned. See [Sentinel setup](#sentinel-setup)
|
|
|
|
documentation for more information.
|
|
|
|
|
|
|
|
All Redis nodes should be configured the same way and with similar server specs, as
|
|
|
|
in a failover situation, any **Slave** can be promoted as the new **Master** by
|
2016-07-19 11:27:50 +00:00
|
|
|
the Sentinel servers.
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
The replication requires authentication, so you need to define a password to
|
|
|
|
protect all Redis nodes and the Sentinels. They will all share the same
|
|
|
|
password, and all instances must be able to talk to
|
2016-10-15 03:40:15 +00:00
|
|
|
each other over the network.
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
### Sentinel setup overview
|
2016-11-10 06:07:30 +00:00
|
|
|
|
2016-11-10 14:51:13 +00:00
|
|
|
Sentinels watch both other sentinels and Redis nodes. Whenever a Sentinel
|
2016-11-10 06:07:30 +00:00
|
|
|
detects that a Redis node is not responding, it will announce that to the
|
2016-11-10 12:14:22 +00:00
|
|
|
other sentinels. You have to reach the **quorum**, the minimum amount of
|
2016-11-10 06:07:30 +00:00
|
|
|
sentinels that agrees that a node is down, to be able to start a failover.
|
|
|
|
|
2016-11-10 12:14:22 +00:00
|
|
|
Whenever the **quorum** is met, you need the **majority** of all known
|
2016-11-10 06:07:30 +00:00
|
|
|
Sentinel nodes to be available and reachable, to elect the Sentinel **leader**
|
|
|
|
who will take all the decisions to restore the service availability by:
|
|
|
|
|
|
|
|
- Promoting a new **Master**
|
|
|
|
- Reconfiguring the other **Slaves** and make them point to the new **Master**
|
|
|
|
- Announce the new **Master** to every other Sentinel peer
|
|
|
|
- Reconfigure the old **Master** and demote to **Slave** when it comes back online
|
|
|
|
|
|
|
|
You must have at least `3` Redis Sentinel servers, and they need to
|
|
|
|
be each in a independent machine (that are believed to fail independently).
|
|
|
|
|
|
|
|
You can configure them in the same machines where you've configured the other
|
|
|
|
Redis servers, but understand that if a whole node goes down, you loose both
|
|
|
|
a Sentinel and a Redis instance.
|
|
|
|
|
|
|
|
The number of sentinels should ideally always be an **odd** number, for the
|
|
|
|
consensus algorithm to be effective in the case of a failure.
|
|
|
|
|
|
|
|
In a `3` nodes topology, you can only afford `1` Sentinel node going down.
|
|
|
|
Whenever the **majority** of the Sentinels goes down, the network partition
|
|
|
|
protection prevents destructive actions and a failover **will not be started**.
|
|
|
|
|
|
|
|
Here are some examples:
|
|
|
|
|
|
|
|
- With `5` or `6` sentinels, a maximum of `2` can go down for a failover begin.
|
|
|
|
- With `7` sentinels, a maximum of `3` nodes can go down.
|
|
|
|
|
|
|
|
The **Leader** election can sometimes fail the voting round when **consensus**,
|
|
|
|
is not achieved (see the odd number of nodes requirement above). In that case,
|
|
|
|
a new attempt will be made after the amount of time defined in
|
|
|
|
`sentinel['failover_timeout']` (in milliseconds).
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
>**Note:**
|
|
|
|
We will see where `sentinel['failover_timeout']` is defined later.
|
|
|
|
|
|
|
|
The `failover_timeout` variable has a lot of different use cases, according to
|
|
|
|
the official documentation:
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
- The time needed to re-start a failover after a previous failover was
|
|
|
|
already tried against the same master by a given Sentinel, is two
|
|
|
|
times the failover timeout.
|
|
|
|
|
|
|
|
- The time needed for a slave replicating to a wrong master according
|
|
|
|
to a Sentinel current configuration, to be forced to replicate
|
|
|
|
with the right master, is exactly the failover timeout (counting since
|
|
|
|
the moment a Sentinel detected the misconfiguration).
|
|
|
|
|
|
|
|
- The time needed to cancel a failover that is already in progress but
|
|
|
|
did not produced any configuration change (SLAVEOF NO ONE yet not
|
|
|
|
acknowledged by the promoted slave).
|
|
|
|
|
|
|
|
- The maximum time a failover in progress waits for all the slaves to be
|
|
|
|
reconfigured as slaves of the new master. However even after this time
|
|
|
|
the slaves will be reconfigured by the Sentinels anyway, but not with
|
|
|
|
the exact parallel-syncs progression as specified.
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
## Redis HA configuration
|
|
|
|
|
|
|
|
This is the section where we install and setup the new Redis instances.
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
>**Notes:**
|
|
|
|
- We assume that you install GitLab and all HA components from scratch. If you
|
|
|
|
already have it installed and running, read how to
|
|
|
|
[switch from a single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha).
|
|
|
|
- Redis nodes (both master and slaves) will need the same password defined in
|
|
|
|
`redis['password']` and `redis['master_password']`. At any time during a
|
|
|
|
failover the Sentinels can reconfigure a node and change its status
|
|
|
|
from master to slave and vice versa.
|
2016-11-10 06:07:30 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
A summary of what are we going to do:
|
2016-11-10 14:51:13 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
1. Provision the required number of instances specified previously:
|
2016-11-10 06:07:30 +00:00
|
|
|
- You can opt to install Redis and Sentinel in the same machine or each in
|
|
|
|
independent ones.
|
2016-11-10 18:00:06 +00:00
|
|
|
- Don't install Redis and Sentinel in the same machines your GitLab application
|
2016-11-10 06:07:30 +00:00
|
|
|
is running on.
|
2016-11-10 12:14:22 +00:00
|
|
|
- All machines must be able to talk to each other and accept incoming
|
2016-11-10 18:00:06 +00:00
|
|
|
connections over Redis (`6379`) and Sentinel (`26379`) ports (unless you
|
|
|
|
change the default ports).
|
2016-11-10 06:07:30 +00:00
|
|
|
- GitLab machines must be able to access these machines and with the same
|
|
|
|
permissions.
|
2016-11-10 14:51:13 +00:00
|
|
|
- Protect them from access from external networks (Internet),
|
2016-11-10 06:07:30 +00:00
|
|
|
to harden the security.
|
|
|
|
|
|
|
|
1. Download/install Omnibus GitLab using **steps 1 and 2** from
|
|
|
|
[GitLab downloads](https://about.gitlab.com/downloads) in each node.
|
|
|
|
- Do not complete other steps on the download page.
|
|
|
|
- Make sure you select the correct Omnibus package, with the same version
|
|
|
|
and type (Community, Enterprise editions) of your current install.
|
|
|
|
|
|
|
|
1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations
|
|
|
|
from running on upgrade. Only the primary GitLab application server should
|
|
|
|
handle migrations.
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
1. Edit `/etc/gitlab/gitlab.rb` and make the changes based on the
|
2016-11-10 06:07:30 +00:00
|
|
|
[Example Configurations](#example-configurations).
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
### Configuring the Master Redis instance
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
You will need to configure the following in `/etc/gitlab/gitlab.rb`:
|
|
|
|
|
|
|
|
1. Define `redis_master_role['enable']` to `true`, to disable other services
|
|
|
|
in the machine (you can still enable Sentinel)
|
|
|
|
|
|
|
|
1. Define a `redis['bind']` address pointing to a local IP that your other machines
|
|
|
|
can reach you.
|
2016-11-10 12:14:22 +00:00
|
|
|
- If you really need to bind to an external accessible IP, make
|
2016-11-10 06:07:30 +00:00
|
|
|
sure you add extra firewall rules to prevent unauthorized access.
|
|
|
|
- You can also set bind to `0.0.0.0` which listen in all interfaces.
|
|
|
|
|
|
|
|
1. Define a `redis['port']` so redis can listen for TCP requests which will
|
|
|
|
allow other machines to connect to it.
|
|
|
|
|
|
|
|
1. Set up a password authentication with `redis['password']` and
|
|
|
|
`redis['master_password']` (use the same password in all nodes).
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
1. [Reconfigure Omnibus GitLab][reocnfigure] for the changes to take effect.
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
### Configuring the Slave Redis instances
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
You will need to configure the following in `/etc/gitlab/gitlab.rb`:
|
|
|
|
|
|
|
|
1. Define `redis_slaves_role['enable']` to `true`, to disable other services
|
|
|
|
in the machine (you can still enable Sentinel)
|
|
|
|
- This will also set automatically `redis['master'] = false`.
|
2016-10-15 03:40:15 +00:00
|
|
|
|
|
|
|
1. Define a `redis['bind']` address pointing to a local IP that your other machines
|
2016-11-10 06:07:30 +00:00
|
|
|
can reach you.
|
2016-11-10 12:14:22 +00:00
|
|
|
- If you really need to bind to an external accessible IP, make
|
2016-10-15 03:40:15 +00:00
|
|
|
sure you add extra firewall rules to prevent unauthorized access.
|
2016-11-10 06:07:30 +00:00
|
|
|
- You can also set bind to `0.0.0.0` which listen in all interfaces.
|
|
|
|
|
2016-10-18 17:45:57 +00:00
|
|
|
1. Define a `redis['port']` so redis can listen for TCP requests which will
|
|
|
|
allow other machines to connect to it.
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
1. Set up a password authentication with `redis['password']` and
|
|
|
|
`redis['master_password']` (use the same password in all nodes).
|
|
|
|
|
|
|
|
1. Define `redis['master_ip']` with the IP of the **Master** Redis.
|
|
|
|
|
|
|
|
1. Define `redis['master_port']` with the port of the **Master** Redis (default to `6379`).
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
Initial **Slave** nodes require `redis['master']` defined to `false` and
|
|
|
|
`redis['master_ip']` pointing to the initial **Master**. If you use the
|
|
|
|
simplified configuration by enabling `redis_slave_role['enable']`, you
|
|
|
|
just need to fill in the `redis['master_ip']`.
|
|
|
|
|
|
|
|
This values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
|
|
|
|
a failover, as the nodes will be managed by the Sentinels, and even after a
|
|
|
|
`gitlab-ctl reconfigure`, they will get their configuration restored by
|
|
|
|
the same Sentinels.
|
|
|
|
|
|
|
|
### Configuring the Sentinel instances
|
|
|
|
|
|
|
|
>**Note:**
|
|
|
|
- Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the
|
|
|
|
Omnibus Community Edition and installations from source, follow the
|
|
|
|
[Redis HA source install](redis_source.md) guide.
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
Now that the Redis servers are all set up, let's configure the Sentinel
|
|
|
|
servers.
|
|
|
|
|
|
|
|
If you are not sure if your Redis servers are working and replicating
|
|
|
|
correctly, please read the [Troubleshooting Replication](#troubleshooting-replication)
|
|
|
|
and fix it before proceeding with Sentinel setup.
|
|
|
|
|
|
|
|
You must have at least `3` Redis Sentinel servers, and they need to
|
|
|
|
be each in a independent machine. You can configure them in the same
|
|
|
|
machines where you've configured the other Redis servers.
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
With GitLab Enterprise Edition, you can use the Omnibus package to setup multiple
|
|
|
|
machines with the Sentinel daemon.
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
See [example configuration](#configuration-for-sentinel-ee-only) below.
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
### Configuring the GitLab application
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
The final part is to inform the main GitLab application server of the Redis
|
|
|
|
Sentinels servers and authentication credentials.
|
|
|
|
|
|
|
|
You can enable or disable Sentinel support at any time in new or existing
|
|
|
|
installations. From the GitLab application perspective, all it requires is
|
|
|
|
the correct credentials for the Sentinel nodes.
|
|
|
|
|
|
|
|
While it doesn't require a list of all Sentinel nodes, in case of a failure,
|
2016-11-10 12:14:22 +00:00
|
|
|
it needs to access at least one of the listed.
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
>**Note:**
|
|
|
|
The following steps should be performed in the [GitLab application server](gitlab.md)
|
2016-11-10 18:00:06 +00:00
|
|
|
which ideally should not have Redis or Sentinels on it for a HA setup.
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:
|
|
|
|
|
|
|
|
- `redis['master_name']` - this is the `master-group-name` from sentinel (default: `gitlab-redis`)
|
|
|
|
- `redis['master_password']` - the same password you've defined before for Redis and Sentinels
|
|
|
|
- `gitlab_rails['redis_sentinels']` - a list of sentinels with `host` and `port`
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
See [example configuration](#configuration-for-gitlab) below.
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
## Switching from an existing single-machine installation to Redis HA
|
|
|
|
|
|
|
|
If you already have a single-machine GitLab install running, you will need to
|
|
|
|
replicate from this machine first, before de-activating the Redis instance
|
|
|
|
inside it.
|
|
|
|
|
|
|
|
Your single-machine install will be the initial **Master**, and the `3` others
|
|
|
|
should be configured as **Slave** pointing to this machine.
|
|
|
|
|
|
|
|
After replication catches up, you will need to stop services in the
|
|
|
|
single-machine install, to rotate the **Master** to one of the new nodes.
|
|
|
|
|
|
|
|
Make the required changes in configuration and restart the new nodes again.
|
|
|
|
|
|
|
|
To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`:
|
|
|
|
|
|
|
|
```ruby
|
|
|
|
redis['enable'] = false
|
|
|
|
```
|
|
|
|
|
|
|
|
If you fail to replicate first, you may loose data (unprocessed background jobs).
|
|
|
|
|
2016-11-10 14:51:13 +00:00
|
|
|
## Minimal example configuration with 1 master, 2 slaves and 3 sentinels
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
In this example we consider that all servers have an internal network
|
|
|
|
interface with IPs in the `10.0.0.x` range, and that they can connect
|
|
|
|
to each other using these IPs.
|
|
|
|
|
|
|
|
In a real world usage, you would also setup firewall rules to prevent
|
2016-11-10 18:00:06 +00:00
|
|
|
unauthorized access from other machines and block traffic from the
|
2016-11-10 06:07:30 +00:00
|
|
|
outside (Internet).
|
|
|
|
|
|
|
|
We will use the same `3` nodes with **Redis** + **Sentinel** topology
|
|
|
|
discussed in [Redis Setup](#redis-setup) and [Sentinel Setup](#sentinel-setup)
|
|
|
|
documentation.
|
|
|
|
|
|
|
|
Here is a list and description of each **machine** and the assigned **IP**:
|
|
|
|
|
|
|
|
* `10.0.0.1`: Redis Master + Sentinel 1
|
|
|
|
* `10.0.0.2`: Redis Slave 1 + Sentinel 2
|
|
|
|
* `10.0.0.2`: Redis Slave 2 + Sentinel 3
|
|
|
|
|
|
|
|
Please note that after the initial configuration, if a failover is initiated
|
|
|
|
by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master**
|
|
|
|
will change permanently (including in `redis.conf`) from one node to the other,
|
|
|
|
until a new failover is initiated again.
|
|
|
|
|
2016-11-10 12:14:22 +00:00
|
|
|
The same thing will happen with `sentinel.conf` that will be overridden after the
|
2016-11-10 06:07:30 +00:00
|
|
|
initial execution, after any new sentinel node starts watching the **Master**,
|
|
|
|
or a failover promotes a different **Master** node.
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
### Configuration for Redis master
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
**Example configation for Redis Master:**
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-10-18 17:45:57 +00:00
|
|
|
In `/etc/gitlab/gitlab.rb`:
|
|
|
|
|
2016-10-15 03:40:15 +00:00
|
|
|
```ruby
|
2016-11-10 06:07:30 +00:00
|
|
|
redis_master_role['enable'] = true
|
|
|
|
|
|
|
|
redis['bind'] = '10.0.0.1'
|
2016-10-15 03:40:15 +00:00
|
|
|
redis['port'] = 6379
|
2016-10-28 22:40:28 +00:00
|
|
|
redis['password'] = 'redis-password-goes-here'
|
2016-10-15 03:40:15 +00:00
|
|
|
redis['master_password'] = 'redis-password-goes-here'
|
|
|
|
```
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
|
|
|
|
|
|
|
|
### Configuration for Redis slaves
|
2016-11-10 06:07:30 +00:00
|
|
|
|
|
|
|
**Example configation for Slave 1:**
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
In `/etc/gitlab/gitlab.rb`:
|
2016-10-15 03:40:15 +00:00
|
|
|
|
|
|
|
```ruby
|
2016-11-10 06:07:30 +00:00
|
|
|
redis_slave_role['enable'] = true
|
|
|
|
|
|
|
|
redis['bind'] = '10.0.0.2'
|
2016-10-15 03:40:15 +00:00
|
|
|
redis['port'] = 6379
|
2016-10-28 22:40:28 +00:00
|
|
|
redis['password'] = 'redis-password-goes-here'
|
2016-10-15 03:40:15 +00:00
|
|
|
redis['master_password'] = 'redis-password-goes-here'
|
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
redis['master_ip'] = '10.0.0.1' # IP of master Redis server
|
|
|
|
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
|
2016-10-15 03:40:15 +00:00
|
|
|
```
|
|
|
|
|
2016-10-18 17:45:57 +00:00
|
|
|
Reconfigure Omnibus GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure`
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
**Example configation for Slave 2:**
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
In `/etc/gitlab/gitlab.rb`:
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
```ruby
|
|
|
|
redis_slave_role['enable'] = true
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
redis['bind'] = '10.0.0.3'
|
|
|
|
redis['port'] = 6379
|
|
|
|
redis['password'] = 'redis-password-goes-here'
|
|
|
|
redis['master_password'] = 'redis-password-goes-here'
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
redis['master_ip'] = '10.0.0.1' # IP of master Redis server
|
|
|
|
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
|
|
|
|
```
|
2016-10-18 17:45:57 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
|
|
|
|
|
|
|
|
### Configuration for Sentinels
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
>**Note:**
|
|
|
|
Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For the
|
|
|
|
Omnibus Community Edition and installations from source, follow the
|
|
|
|
[Redis HA source install](redis_source.md) guide.
|
2016-08-02 01:25:28 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
Please note that some of the variables are already configured previously
|
|
|
|
as they are required for Redis replication.
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
**Example configation for Sentinel 1:**
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
In `/etc/gitlab/gitlab.rb`:
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
```ruby
|
|
|
|
redis_sentinel_role['enable'] = true
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
|
2016-11-10 18:00:06 +00:00
|
|
|
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
|
2016-11-10 06:07:30 +00:00
|
|
|
redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance
|
|
|
|
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
## Configure Sentinel
|
|
|
|
sentinel['bind'] = '10.0.0.1'
|
|
|
|
# sentinel['port'] = 26379 # uncomment to change default port
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
## Quorum must reflect the amount of voting sentinels it take to start a failover.
|
|
|
|
## Value must NOT be greater then the ammount of sentinels.
|
|
|
|
##
|
|
|
|
## The quorum can be used to tune Sentinel in two ways:
|
|
|
|
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
|
|
|
|
## we deploy, we are basically making Sentinel more sensible to master failures,
|
|
|
|
## triggering a failover as soon as even just a minority of Sentinels is no longer
|
|
|
|
## able to talk with the master.
|
|
|
|
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
|
|
|
|
## making Sentinel able to failover only when there are a very large number (larger
|
|
|
|
## than majority) of well connected Sentinels which agree about the master being down.s
|
|
|
|
sentinel['quorum'] = 2
|
2016-11-08 05:24:43 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
## Consider unresponsive server down after x amount of ms.
|
|
|
|
# sentinel['down_after_milliseconds'] = 10000
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
## Specifies the failover timeout in milliseconds. It is used in many ways:
|
|
|
|
##
|
|
|
|
## - The time needed to re-start a failover after a previous failover was
|
|
|
|
## already tried against the same master by a given Sentinel, is two
|
|
|
|
## times the failover timeout.
|
|
|
|
##
|
|
|
|
## - The time needed for a slave replicating to a wrong master according
|
|
|
|
## to a Sentinel current configuration, to be forced to replicate
|
|
|
|
## with the right master, is exactly the failover timeout (counting since
|
|
|
|
## the moment a Sentinel detected the misconfiguration).
|
|
|
|
##
|
|
|
|
## - The time needed to cancel a failover that is already in progress but
|
|
|
|
## did not produced any configuration change (SLAVEOF NO ONE yet not
|
|
|
|
## acknowledged by the promoted slave).
|
|
|
|
##
|
|
|
|
## - The maximum time a failover in progress waits for all the slaves to be
|
|
|
|
## reconfigured as slaves of the new master. However even after this time
|
|
|
|
## the slaves will be reconfigured by the Sentinels anyway, but not with
|
|
|
|
## the exact parallel-syncs progression as specified.
|
|
|
|
# sentinel['failover_timeout'] = 60000
|
|
|
|
```
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
**Example configation for Sentinel 2:**
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
In `/etc/gitlab/gitlab.rb`:
|
2016-10-15 03:40:15 +00:00
|
|
|
|
|
|
|
```ruby
|
2016-11-10 06:07:30 +00:00
|
|
|
redis_sentinel_role['enable'] = true
|
|
|
|
|
|
|
|
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
|
2016-11-10 18:00:06 +00:00
|
|
|
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
|
2016-11-10 06:07:30 +00:00
|
|
|
redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance
|
|
|
|
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
## Configure Sentinel
|
|
|
|
sentinel['bind'] = '10.0.0.2'
|
|
|
|
# sentinel['port'] = 26379 # uncomment to change default port
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
## Quorum must reflect the amount of voting sentinels it take to start a failover.
|
|
|
|
## Value must NOT be greater then the ammount of sentinels.
|
|
|
|
##
|
|
|
|
## The quorum can be used to tune Sentinel in two ways:
|
|
|
|
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
|
|
|
|
## we deploy, we are basically making Sentinel more sensible to master failures,
|
|
|
|
## triggering a failover as soon as even just a minority of Sentinels is no longer
|
|
|
|
## able to talk with the master.
|
|
|
|
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
|
|
|
|
## making Sentinel able to failover only when there are a very large number (larger
|
|
|
|
## than majority) of well connected Sentinels which agree about the master being down.s
|
|
|
|
sentinel['quorum'] = 2
|
|
|
|
|
|
|
|
## Consider unresponsive server down after x amount of ms.
|
|
|
|
# sentinel['down_after_milliseconds'] = 10000
|
|
|
|
|
|
|
|
## Specifies the failover timeout in milliseconds. It is used in many ways:
|
|
|
|
##
|
|
|
|
## - The time needed to re-start a failover after a previous failover was
|
|
|
|
## already tried against the same master by a given Sentinel, is two
|
|
|
|
## times the failover timeout.
|
|
|
|
##
|
|
|
|
## - The time needed for a slave replicating to a wrong master according
|
|
|
|
## to a Sentinel current configuration, to be forced to replicate
|
|
|
|
## with the right master, is exactly the failover timeout (counting since
|
|
|
|
## the moment a Sentinel detected the misconfiguration).
|
|
|
|
##
|
|
|
|
## - The time needed to cancel a failover that is already in progress but
|
|
|
|
## did not produced any configuration change (SLAVEOF NO ONE yet not
|
|
|
|
## acknowledged by the promoted slave).
|
|
|
|
##
|
|
|
|
## - The maximum time a failover in progress waits for all the slaves to be
|
|
|
|
## reconfigured as slaves of the new master. However even after this time
|
|
|
|
## the slaves will be reconfigured by the Sentinels anyway, but not with
|
|
|
|
## the exact parallel-syncs progression as specified.
|
|
|
|
# sentinel['failover_timeout'] = 60000
|
|
|
|
```
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
**Example configation for Sentinel 3:**
|
|
|
|
|
|
|
|
In `/etc/gitlab/gitlab.rb`:
|
|
|
|
|
|
|
|
```ruby
|
2016-10-28 22:40:28 +00:00
|
|
|
redis_sentinel_role['enable'] = true
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-10-18 17:45:57 +00:00
|
|
|
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
|
2016-11-10 18:00:06 +00:00
|
|
|
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
|
2016-10-18 17:45:57 +00:00
|
|
|
redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance
|
2016-11-10 06:07:30 +00:00
|
|
|
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
|
2016-10-18 17:45:57 +00:00
|
|
|
|
|
|
|
## Configure Sentinel
|
2016-11-10 06:07:30 +00:00
|
|
|
sentinel['bind'] = '10.0.0.3'
|
2016-10-18 17:45:57 +00:00
|
|
|
# sentinel['port'] = 26379 # uncomment to change default port
|
|
|
|
|
|
|
|
## Quorum must reflect the amount of voting sentinels it take to start a failover.
|
|
|
|
## Value must NOT be greater then the ammount of sentinels.
|
|
|
|
##
|
|
|
|
## The quorum can be used to tune Sentinel in two ways:
|
|
|
|
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
|
|
|
|
## we deploy, we are basically making Sentinel more sensible to master failures,
|
|
|
|
## triggering a failover as soon as even just a minority of Sentinels is no longer
|
|
|
|
## able to talk with the master.
|
|
|
|
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
|
|
|
|
## making Sentinel able to failover only when there are a very large number (larger
|
|
|
|
## than majority) of well connected Sentinels which agree about the master being down.s
|
|
|
|
sentinel['quorum'] = 2
|
|
|
|
|
|
|
|
## Consider unresponsive server down after x amount of ms.
|
|
|
|
# sentinel['down_after_milliseconds'] = 10000
|
|
|
|
|
|
|
|
## Specifies the failover timeout in milliseconds. It is used in many ways:
|
|
|
|
##
|
|
|
|
## - The time needed to re-start a failover after a previous failover was
|
|
|
|
## already tried against the same master by a given Sentinel, is two
|
|
|
|
## times the failover timeout.
|
|
|
|
##
|
|
|
|
## - The time needed for a slave replicating to a wrong master according
|
|
|
|
## to a Sentinel current configuration, to be forced to replicate
|
|
|
|
## with the right master, is exactly the failover timeout (counting since
|
|
|
|
## the moment a Sentinel detected the misconfiguration).
|
|
|
|
##
|
|
|
|
## - The time needed to cancel a failover that is already in progress but
|
|
|
|
## did not produced any configuration change (SLAVEOF NO ONE yet not
|
|
|
|
## acknowledged by the promoted slave).
|
|
|
|
##
|
|
|
|
## - The maximum time a failover in progress waits for all the slaves to be
|
|
|
|
## reconfigured as slaves of the new master. However even after this time
|
|
|
|
## the slaves will be reconfigured by the Sentinels anyway, but not with
|
|
|
|
## the exact parallel-syncs progression as specified.
|
|
|
|
# sentinel['failover_timeout'] = 60000
|
|
|
|
```
|
2016-10-15 03:40:15 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
## Advanced configuration
|
|
|
|
|
|
|
|
Omnibus GitLab configures some things behind the curtains to make the sysadmins'
|
|
|
|
lives easier. If you want to know what happens underneath keep reading.
|
|
|
|
|
2016-11-10 06:07:30 +00:00
|
|
|
### Control running services
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
In the previous example above we've used `redis_sentinel_role` and
|
|
|
|
`redis_master_role` which simplify the amount of configuration changes.
|
2016-10-28 22:40:28 +00:00
|
|
|
|
|
|
|
If you want more control, here is what each one sets for you automatically
|
|
|
|
when enabled:
|
|
|
|
|
|
|
|
```ruby
|
|
|
|
## Redis Sentinel Role
|
|
|
|
redis_sentinel_role['enable'] = true
|
|
|
|
|
|
|
|
# When Sentinel Role is enabled, the following services are enabled/disabled:
|
|
|
|
sentinel['enable'] = true
|
|
|
|
|
|
|
|
# This others are disabled:
|
|
|
|
redis['enable'] = false
|
|
|
|
bootstrap['enable'] = false
|
|
|
|
nginx['enable'] = false
|
|
|
|
postgresql['enable'] = false
|
|
|
|
gitlab_rails['enable'] = false
|
|
|
|
mailroom['enable'] = false
|
|
|
|
|
|
|
|
## Redis master/slave Role:
|
|
|
|
redis_master_role['enable'] = true # enable only one of them
|
|
|
|
redis_slave_role['enable'] = true # enable only one of them
|
|
|
|
|
|
|
|
# When Redis Master or Slave role are enabled, the following services are enabled/disabled:
|
|
|
|
# (Note that if redis and sentinel roles are combined both services will be enabled)
|
|
|
|
|
|
|
|
# When Sentinel Role is enabled, the following services are enabled/disabled:
|
|
|
|
redis['enable'] = true
|
|
|
|
|
|
|
|
# This others are disabled:
|
|
|
|
sentinel['enable'] = false
|
|
|
|
bootstrap['enable'] = false
|
|
|
|
nginx['enable'] = false
|
|
|
|
postgresql['enable'] = false
|
|
|
|
gitlab_rails['enable'] = false
|
|
|
|
mailroom['enable'] = false
|
|
|
|
|
|
|
|
# Redis Slave role also change this setting from default 'true' to 'false':
|
|
|
|
redis['master'] = false
|
|
|
|
```
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
You can find the relevant attributes defined in [gitlab_rails.rb][omnifile].
|
|
|
|
|
2016-10-18 17:45:57 +00:00
|
|
|
## Troubleshooting
|
|
|
|
|
|
|
|
There are a lot of moving parts that needs to be taken care carefully
|
|
|
|
in order for the HA setup to work as expected.
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
Before proceeding with the troubleshooting below, check your firewall rules:
|
|
|
|
|
2016-10-18 17:45:57 +00:00
|
|
|
- Redis machines
|
|
|
|
- Accept TCP connection in `6379`
|
|
|
|
- Connect to the other Redis machines via TCP in `6379`
|
|
|
|
- Sentinel machines
|
|
|
|
- Accept TCP connection in `26379`
|
|
|
|
- Connect to other Sentinel machines via TCP in `26379`
|
|
|
|
- Connect to the Redis machines via TCP in `6379`
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
### Troubleshooting Redis replication
|
2016-10-18 17:45:57 +00:00
|
|
|
|
|
|
|
You can check if everything is correct by connecting to each server using
|
|
|
|
`redis-cli` application, and sending the `INFO` command.
|
|
|
|
|
|
|
|
If authentication was correctly defined, it should fail with:
|
|
|
|
`NOAUTH Authentication required` error. Try to authenticate with the
|
|
|
|
previous defined password with `AUTH redis-password-goes-here` and
|
|
|
|
try the `INFO` command again.
|
|
|
|
|
|
|
|
Look for the `# Replication` section where you should see some important
|
|
|
|
information like the `role` of the server.
|
|
|
|
|
|
|
|
When connected to a `master` redis, you will see the number of connected
|
|
|
|
`slaves`, and a list of each with connection details:
|
|
|
|
|
|
|
|
```
|
|
|
|
# Replication
|
|
|
|
role:master
|
|
|
|
connected_slaves:1
|
|
|
|
slave0:ip=10.133.5.21,port=6379,state=online,offset=208037514,lag=1
|
|
|
|
master_repl_offset:208037658
|
|
|
|
repl_backlog_active:1
|
|
|
|
repl_backlog_size:1048576
|
|
|
|
repl_backlog_first_byte_offset:206989083
|
|
|
|
repl_backlog_histlen:1048576
|
|
|
|
```
|
|
|
|
|
|
|
|
When it's a `slave`, you will see details of the master connection and if
|
|
|
|
its `up` or `down`:
|
|
|
|
|
|
|
|
```
|
|
|
|
# Replication
|
|
|
|
role:slave
|
|
|
|
master_host:10.133.1.58
|
|
|
|
master_port:6379
|
|
|
|
master_link_status:up
|
|
|
|
master_last_io_seconds_ago:1
|
|
|
|
master_sync_in_progress:0
|
|
|
|
slave_repl_offset:208096498
|
|
|
|
slave_priority:100
|
|
|
|
slave_read_only:1
|
|
|
|
connected_slaves:0
|
|
|
|
master_repl_offset:0
|
|
|
|
repl_backlog_active:0
|
|
|
|
repl_backlog_size:1048576
|
|
|
|
repl_backlog_first_byte_offset:0
|
|
|
|
repl_backlog_histlen:0
|
|
|
|
```
|
|
|
|
|
2016-11-10 18:00:06 +00:00
|
|
|
### Troubleshooting Sentinel
|
2016-10-06 23:47:22 +00:00
|
|
|
|
|
|
|
If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
|
|
|
|
there may be something wrong with your configuration files or it can be related
|
|
|
|
to [this issue][gh-531].
|
|
|
|
|
|
|
|
You must make sure you are defining the same value in `redis['master_name']`
|
|
|
|
and `redis['master_pasword']` as you defined for your sentinel node.
|
|
|
|
|
|
|
|
The way the redis connector `redis-rb` works with sentinel is a bit
|
|
|
|
non-intuitive. We try to hide the complexity in omnibus, but it still requires
|
|
|
|
a few extra configs.
|
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
---
|
2016-04-22 18:18:06 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
To make sure your configuration is correct:
|
2016-04-22 18:18:06 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
1. SSH into your GitLab application server
|
|
|
|
1. Enter the Rails console:
|
|
|
|
|
|
|
|
```
|
|
|
|
# For Omnibus installations
|
|
|
|
sudo gitlab-rails console
|
|
|
|
|
|
|
|
# For source installations
|
2016-10-15 03:40:15 +00:00
|
|
|
sudo -u git rails console production
|
2016-07-19 11:27:50 +00:00
|
|
|
```
|
|
|
|
|
|
|
|
1. Run in the console:
|
2016-04-22 18:18:06 +00:00
|
|
|
|
|
|
|
```ruby
|
2016-07-19 11:27:50 +00:00
|
|
|
redis = Redis.new(Gitlab::Redis.params)
|
|
|
|
redis.info
|
2016-04-22 18:18:06 +00:00
|
|
|
```
|
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
Keep this screen open and try to simulate a failover below.
|
2016-04-22 18:18:06 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
1. To simulate a failover on master Redis, SSH into the Redis server and run:
|
|
|
|
|
|
|
|
```bash
|
2016-11-08 05:24:43 +00:00
|
|
|
# port must match your master redis port, and the sleep time must be a few seconds bigger than defined one
|
|
|
|
redis-cli -h localhost -p 6379 DEBUG sleep 20
|
2016-07-19 11:27:50 +00:00
|
|
|
```
|
|
|
|
|
|
|
|
1. Then back in the Rails console from the first step, run:
|
|
|
|
|
|
|
|
```
|
|
|
|
redis.info
|
|
|
|
```
|
|
|
|
|
|
|
|
You should see a different port after a few seconds delay
|
|
|
|
(the failover/reconnect time).
|
2016-04-22 18:18:06 +00:00
|
|
|
|
2016-11-10 12:21:34 +00:00
|
|
|
|
|
|
|
## Changelog
|
|
|
|
|
|
|
|
Changes to Redis HA over time.
|
|
|
|
|
|
|
|
### Experimental Redis Sentinel support
|
|
|
|
|
|
|
|
>
|
|
|
|
Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11.
|
|
|
|
Starting with 8.14, Redis Sentinel is no longer experimental.
|
|
|
|
If you used with versions `< 8.14` before, please check the updated
|
|
|
|
documentation here.
|
|
|
|
|
2016-04-22 18:18:06 +00:00
|
|
|
---
|
|
|
|
|
|
|
|
Read more on high-availability configuration:
|
|
|
|
|
|
|
|
1. [Configure the database](database.md)
|
|
|
|
1. [Configure NFS](nfs.md)
|
|
|
|
1. [Configure the GitLab application servers](gitlab.md)
|
|
|
|
1. [Configure the load balancers](load_balancer.md)
|
2016-07-19 11:27:50 +00:00
|
|
|
|
|
|
|
[ce-1877]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1877
|
|
|
|
[restart]: ../restart_gitlab.md#installations-from-source
|
|
|
|
[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure
|
|
|
|
[gh-531]: https://github.com/redis/redis-rb/issues/531
|
|
|
|
[gh-534]: https://github.com/redis/redis-rb/issues/534
|
2016-11-10 14:51:13 +00:00
|
|
|
[redis]: http://redis.io/
|
|
|
|
[sentinel]: http://redis.io/topics/sentinel
|
2016-11-10 18:00:06 +00:00
|
|
|
[omnifile]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/82b7345b150f072c8673c79738ce893f92d0d652/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb#L134-159
|
|
|
|
[source]: ../../install/installation.md
|
|
|
|
[ce]: https://about.gitlab.com/downloads
|
|
|
|
[ee]: https://about.gitlab.com/downloads-ee
|