diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index bfad3047385..f49bf1d4ab5 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -19,7 +19,6 @@ that comes bundled with Omnibus GitLab packages. - [Prerequisites](#prerequisites) - [Redis setup](#redis-setup) - [Existing single-machine installation](#existing-single-machine-installation) - - [Installation from source](#installation-from-source) - [Omnibus packages](#omnibus-packages) - [Configuring Sentinel](#configuring-sentinel) - [How sentinel handles a failover](#how-sentinel-handles-a-failover) @@ -31,7 +30,6 @@ that comes bundled with Omnibus GitLab packages. - [Redis replication](#redis-replication) - [Sentinel](#sentinel) - [Omnibus GitLab](#omnibus-gitlab) - - [Install from Source](#install-from-source) @@ -161,58 +159,6 @@ To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`: redis['enable'] = false ``` -#### Installation from source - -**Configuring Master Redis instance** - -You need to make the following changes in `redis.conf`: - -1. Define a `bind` address pointing to a local IP that your other machines - can reach you. If you really need to bind to an external acessible IP, make - sure you add extra firewall rules to prevent unauthorized access: - - ```conf - # By default, if no "bind" configuration directive is specified, Redis listens - # for connections from all the network interfaces available on the server. - # It is possible to listen to just one or multiple selected interfaces using - # the "bind" configuration directive, followed by one or more IP addresses. - # - # Examples: - # - # bind 192.168.1.100 10.0.0.1 - # bind 127.0.0.1 ::1 - bind 0.0.0.0 # This will bind to all interfaces - ``` - -1. Define a `port` to force redis to listin on TCP so other machines can - connect to it: - - ```conf - # Accept connections on the specified port, default is 6379 (IANA #815344). - # If port 0 is specified Redis will not listen on a TCP socket. - port 6379 - ``` - -1. Set up password authentication (use the same password in all nodes) - - ```conf - requirepass "redis-password-goes-here" - masterauth "redis-password-goes-here" - ``` - -1. Restart the Redis services for the changes to take effect. - -**Configuring Slave Redis instance** - -1. Follow same instructions from master, with the extra change in `redis.conf`: - - ```conf - # IP and port of the master Redis server - slaveof 10.10.10.10 6379 - ``` - -1. Restart the Redis services for the changes to take effect. - #### Omnibus packages You need to install the Omnibus GitLab package in `3` independent machines. @@ -304,25 +250,16 @@ GitLab Enterprise Edition provides [automated way to setup and run](#sentinel-se #### Sentinel setup ##### Community Edition + With GitLab Community Edition, you need to install, configure, execute and monitor Sentinel from source. Omnibus GitLab Community Edition package does not support Sentinel configuration. -A minimal configuration file (`sentinel.conf`) should contain the following: - -```conf -bind 0.0.0.0 # bind to all interfaces or change to a specific IP -port 26379 # default sentinel port -sentinel auth-pass gitlab-redis redis-password-goes-here -sentinel monitor gitlab-redis 10.0.0.1 6379 2 -sentinel down-after-milliseconds gitlab-redis 10000 -sentinel config-epoch gitlab-redis 0 -sentinel leader-epoch gitlab-redis 0 -``` +See documentation for Source Install [here](redis_source.md). ##### Enterprise Edition -To setup sentinel, you edit `/etc/gitlab/gitlab.rb` file: +To setup sentinel, edit `/etc/gitlab/gitlab.rb` file: ```ruby @@ -336,7 +273,7 @@ redis_master_role['enable'] = true ## Enabled Sentinel and Redis Slave services redis_sentinel_role['enable'] = true -redis_master_role['enable'] = true +redis_slave_role['enable'] = true ## Configure Redis redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node @@ -345,7 +282,7 @@ redis['master_port'] = 6379 # port of the initial master redis instance redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance ## Configure Sentinel -sentinel['bind'] = '0.0.0.0' # or specify an IP to bind to a single one +# sentinel['bind'] = '0.0.0.0' # bind to all interfaces, uncomment to specify an IP and bind to a single one # sentinel['port'] = 26379 # uncomment to change default port ## Quorum must reflect the amount of voting sentinels it take to start a failover. @@ -435,24 +372,16 @@ master and the new sentinels servers. ### GitLab setup -You can enable or disable sentinel support at any time in new or existing +You can enable or disable Sentinel support at any time in new or existing installations. From the GitLab application perspective, all it requires is -the correct credentials for the master Redis and for all Sentinel nodes. +the correct credentials for the Sentinel nodes. -It doesn't require a list of all Sentinel nodes, as in case of a failure, -the application will need to query only one of them. +While it doesn't require a list of all Sentinel nodes, in case of a failure, +it needs to access at one of listed ones. >**Note:** -The following steps should be performed in the [GitLab application server](gitlab.md). - -**For source based installations** - -1. Edit `/home/git/gitlab/config/resque.yml` following the example in - `/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels - line, changing to the correct server credentials. -1. Restart GitLab for the changes to take effect. - -**For Omnibus installations** +The following steps should be performed in the [GitLab application server](gitlab.md) +which ideally should not have Redis or Sentinels in the same machine for a HA setup. 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: @@ -466,7 +395,7 @@ The following steps should be performed in the [GitLab application server](gitla ] ``` -1. [Reconfigure] the GitLab for the changes to take effect. +1. [Reconfigure] GitLab for the changes to take effect. ## Troubleshooting @@ -548,42 +477,6 @@ The way the redis connector `redis-rb` works with sentinel is a bit non-intuitive. We try to hide the complexity in omnibus, but it still requires a few extra configs. -#### Install from Source - -If you get an error like: `Redis::CannotConnectError: No sentinels available.`, -there may be something wrong with your configuration files or it can be related -to [this issue][gh-531]. - -It's a bit non-intuitive the way you have to config `resque.yml` and -`sentinel.conf`, otherwise `redis-rb` will not work properly. - -The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`) -**must** be used as the hostname in GitLab (`resque.yml` for source installations -or `gitlab-rails['redis_*']` in Omnibus): - -```conf -# sentinel.conf: -sentinel monitor gitlab-redis 10.10.10.10 6379 2 -sentinel down-after-milliseconds gitlab-redis 10000 -sentinel config-epoch gitlab-redis 0 -sentinel leader-epoch gitlab-redis 0 -``` - -```yaml -# resque.yaml -production: - url: redis://:myredispassword@gitlab-redis/ - sentinels: - - - host: slave1.example.com # or use ip - port: 26380 # point to sentinel, not to redis port - - - host: slave2.exampl.com # or use ip - port: 26381 # point to sentinel, not to redis port -``` - -When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel) - --- To make sure your configuration is correct: @@ -611,8 +504,8 @@ To make sure your configuration is correct: 1. To simulate a failover on master Redis, SSH into the Redis server and run: ```bash - # port must match your master redis port - redis-cli -h localhost -p 6379 DEBUG sleep 60 + # port must match your master redis port, and the sleep time must be a few seconds bigger than defined one + redis-cli -h localhost -p 6379 DEBUG sleep 20 ``` 1. Then back in the Rails console from the first step, run: diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md new file mode 100644 index 00000000000..7e8c8c2b4b9 --- /dev/null +++ b/doc/administration/high_availability/redis_source.md @@ -0,0 +1,286 @@ +# Configuring Redis for GitLab HA (Source Install) + +We highly recommend that you use Omnibus GitLab packages, as we can optimize +required packages specifically for GitLab, and we will take care of upgrading +to the latest supported version. + +If you are building packages for a specific distro, or trying to build some +internal automation, you can check this documentation to learn about the +minimal setup, required changes, etc. + +If you want to see the documentation for Omnibus GitLab Install, please [read it here](redis.md). + + + +**Table of Contents** + +- [Configure your own Redis server](#configure-your-own-redis-server) + - [Configuring Master Redis instance](#configuring-master-redis-instance) + - [Configuring Slave Redis instances](#configuring-slave-redis-instances) + - [Configuring Redis Sentinel instances](#configuring-redis-sentinel-instances) +- [GitLab setup](#gitlab-setup) +- [Example configurations](#example-configurations) + - [Configuring Redis Master](#configuring-redis-master) + - [Configuring Redis Slaves](#configuring-redis-slaves) + - [Configuring Redis Sentinel](#configuring-redis-sentinel) +- [Troubleshooting](#troubleshooting) + + + +## Configure your own Redis server + +Redis server must be configured to use TCP connection instead of socket, +and since Redis `3.2`, you must define a password to receive external +connections (`requirepass`). + +You will also need to define equal password for slave password definition +(`masterauth`), in the same instance, if you are using Redis with Sentinel. + +To configure Redis to use TCP connection you need to define both +`bind` and `port`. You can bind to all interfaces (`0.0.0.0`) or specify the +ip of the desired interface (for ex. one from an internal network). + + +### Configuring Master Redis instance + +You need to make the following changes in `redis.conf`: + +1. Define a `bind` address pointing to a local IP that your other machines + can reach you. If you really need to bind to an external acessible IP, make + sure you add extra firewall rules to prevent unauthorized access: + +1. Define a `port` to force redis to listen on TCP so other machines can + connect to it (default port is `6379`). + +1. Set up password authentication (use the same password in all nodes). + The password should be defined equal for both `requirepass` and `masterauth` + when setting up Redis to use with Sentinel. + +1. Restart the Redis services for the changes to take effect. + +See [example configuration](#configuring-redis-master) below. + +### Configuring Slave Redis instances + +1. Follow same instructions for Redis Master + +1. Define `slaveof` pointing to the Redis master instance with **IP** and **port**. + +1. Restart the Redis services for the changes to take effect. + +See [example configuration](#configuring-redis-slaves) below. + +### Configuring Redis Sentinel instances + +Sentinel is a special type of Redis server. It inherits most of the basic +configuration options you can define in `redis.conf`, with specific ones +starting with `sentinel` prefix. + +You will need to define the initial configs to enable connectivity: + +1. Define a `bind` address pointing to a local IP that your other machines + can reach you. If you really need to bind to an external acessible IP, make + sure you add extra firewall rules to prevent unauthorized access: + +1. Define a `port` to force sentinel to listen on TCP so other machines can + connect to it (default port is `26379`). + +And the sentinel specific ones: + +1. Define with `sentinel auth-pass` the same shared password you have + defined for both Redis **Master** and **Slaves** instances. + +1. Define with `sentinel monitor` the **IP** and **port** of the Redis + **Master** node, and the **quorum** required to start a failover. + If you need more information to understand about quorum, please + read the detailed explanation in the [HA documentation for Omnibus Installs](redis.md). + +1. Define with `sentinel down-after-milliseconds` the ammount in `ms` of time + that an unresponsive server will be considered down. + +1. Define a value for `sentinel failover_timeout` in `ms`. This has multiple + meanings: + + * The time needed to re-start a failover after a previous failover was + already tried against the same master by a given Sentinel, is two + times the failover timeout. + + * The time needed for a slave replicating to a wrong master according + to a Sentinel current configuration, to be forced to replicate + with the right master, is exactly the failover timeout (counting since + the moment a Sentinel detected the misconfiguration). + + * The time needed to cancel a failover that is already in progress but + did not produced any configuration change (SLAVEOF NO ONE yet not + acknowledged by the promoted slave). + + * The maximum time a failover in progress waits for all the slaves to be + reconfigured as slaves of the new master. However even after this time + the slaves will be reconfigured by the Sentinels anyway, but not with + the exact parallel-syncs progression as specified. + +See [example configuration](#configuring-redis-sentinel) below. + +## GitLab setup + +You can enable or disable Sentinel support at any time in new or existing +installations. From the GitLab application perspective, all it requires is +the correct credentials for the Sentinel nodes. + +While it doesn't require a list of all Sentinel nodes, in case of a failure, +it needs to access at one of listed ones. + +>**Note:** +The following steps should be performed in the [GitLab application server](gitlab.md) +which ideally should not have Redis or Sentinels in the same machine for a HA setup. + +1. Edit `/home/git/gitlab/config/resque.yml` following the example in + `/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels + lines, pointing to the correct server credentials. + +1. Restart GitLab for the changes to take effect. + +## Example configurations + +In this example we consider that all servers have an internal network +interface with IPs in the `10.0.0.x` range, and that they can connect +to each other using these IPs. + +In a real world usage, you would also setup firewall rules to prevent +unauthorized access from other machines, and block traffic from the +outside (Internet). + +We will use the same `3` nodes with **Redis** + **Sentinel** topology +discussed in the [Configuring Redis for GitLab HA](redis.md) documentation. + +Here is a list and description of each **machine** and the assined **ip**: + +* `10.0.0.1`: Redis Master + Sentinel 1 +* `10.0.0.2`: Redis Slave 1 + Sentinel 2 +* `10.0.0.2`: Redis Slave 2 + Sentinel 3 + +Please note that after the initial configuration, if a failover is initiated +by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master** +will change permanently (including in `redis.conf`) from one node to the other, +until a new failover is initiated again. + +The same thing will happen with `sentinel.conf` that will be overriten after the +initial execution, after any new sentinel node starts watching the **Master**, +or a failover promotes a different **Master** node. + +### Configuring Redis Master + +`redis.conf`: + +```conf +bind 10.0.0.1 +port 6379 +requirepass redis-password-goes-here +masterauth redis-password-goes-here +``` + +### Configuring Redis Slaves + +**Slave 1 - `redis.conf`:** + +```conf +bind 10.0.0.2 +port 6379 +requirepass redis-password-goes-here +masterauth redis-password-goes-here + +# IP and port of the master Redis server +slaveof 10.0.0.1 6379 +``` + +**Slave 2 - `redis.conf`:** + +```conf +bind 10.0.0.3 +port 6379 +requirepass redis-password-goes-here +masterauth redis-password-goes-here + +# IP and port of the master Redis server +slaveof 10.0.0.1 6379 +``` + +### Configuring Redis Sentinel + +For this example, **Sentinel 1** will be configured in the same machine as the +**Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the +**Slave 1** and **Slave 2** respectively. + +Sentinel 1 - `sentinel.conf` + +```conf +bind 10.0.0.1 +port 26379 +sentinel auth-pass gitlab-redis redis-password-goes-here +sentinel monitor gitlab-redis 10.0.0.1 6379 2 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel failover_timeout 30000 +``` + +Sentinel 2 - `sentinel.conf` + +```conf +bind 10.0.0.2 +port 26379 +sentinel auth-pass gitlab-redis redis-password-goes-here +sentinel monitor gitlab-redis 10.0.0.1 6379 2 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel failover_timeout 30000 +``` + +Sentinel 3 - `sentinel.conf` + +```conf +bind 10.0.0.3 +port 26379 +sentinel auth-pass gitlab-redis redis-password-goes-here +sentinel monitor gitlab-redis 10.0.0.1 6379 2 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel failover_timeout 30000 +``` + +## Troubleshooting + +We have a more detailed [Troubleshooting](redis.md#troubleshooting) explained in the documentation for Omnibus +Install. Here we will list only the things that are specific to a **Source** install. + +If you get an error in GitLab like: `Redis::CannotConnectError: No sentinels available.`, +there may be something wrong with your configuration files or it can be related +to [this issue][gh-531]. + +It's a bit non-intuitive the way you have to config `resque.yml` and +`sentinel.conf`, otherwise `redis-rb` will not work properly. + +The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`) +**must** be used as the hostname in GitLab (`resque.yml`): + +```conf +# sentinel.conf: +sentinel monitor gitlab-redis 10.0.0.1 6379 2 +sentinel down-after-milliseconds gitlab-redis 10000 +sentinel config-epoch gitlab-redis 0 +sentinel leader-epoch gitlab-redis 0 +``` + +```yaml +# resque.yaml +production: + url: redis://:myredispassword@gitlab-redis/ + sentinels: + - + host: 10.0.0.1 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.2 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.3 + port: 26379 # point to sentinel, not to redis port +``` + +When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel)