diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 1adc681e98b..8a6c5869e70 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -1,114 +1,12 @@ # Configuring Redis for GitLab HA -You can choose to install and manage Redis yourself, or you can use GitLab -Omnibus packages to help. +You can choose to install and manage Redis yourself, or you can use the one +that comes bundled with GitLab Omnibus packages. -## Experimental Redis Sentinel support - -Since 8.10 release, you can configure a list of Redis Sentinel servers that -will monitor a group of Redis servers to provide you with a standard failover -support. - -There is currently one exception to the Sentinel support: **mail_room**, the -component that process incoming emails. - -It doesn't support Sentinel yet, but we hope to integrate a future release -that does support it. - -To get a better understanding on how to correctly setup Sentinel, please read -[Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as -faling to configure it correctly can lead to data-loss. - -### Redis setup - -You must have at least 2 Redis servers: 1 Master, 1 or more Slaves. -They should be configured the same way and with similar server specs, as -in a failover situation, any Slave can be elected as the new Master by -the Sentinels servers. - -In a minimal setup, the only required change for the slaves in `redis.conf` -is the addition of a `slaveof` line pointing to the initial master like this: - -```conf -slaveof 192.168.1.1 6379 -``` - -You can increase the security by defining a `requirepass` configuration in -the master: - -```conf -requirepass " -``` - -and adding this line to all the slave servers: - -```conf -masterauth "" -``` - -> **Note** This setup is not safe to be used by a machine accessible by the -internet. Use it in combination with tight firewall rules. - -### Sentinel setup - -The support for Sentinel in ruby have some [caveats](https://github.com/redis/redis-rb/issues/531). -While you can give any name for the `master-group-name` part of the -configuration, as in this example: - -```conf -sentinel monitor ` -``` - -For it to work in ruby, you have to use the "hostname" of the master redis -server otherwhise you will get an error message like this one: -`Redis::CannotConnectError: No sentinels available.`. - - -Here is an example configuration file (`sentinel.conf`) for a Sentinel node: - -```conf -port 26379 -sentinel monitor master-redis.example.com 10.10.10.10 6379 1 -sentinel down-after-milliseconds master-redis.example.com 10000 -sentinel config-epoch master-redis.example.com 0 -sentinel leader-epoch locmaster-redis.example.comalhost 0 -``` - -### GitLab setup - -You can enable or disable sentinel support at any time in new or existing -installs. From the GitLab application perspective, all it requires is -the correct credentials for the Master redis and for a few Sentinels nodes. - -It doesn't require a list of all sentinel nodes, as in case of a failure, -the application will need to query only one of them. - -For a source based install, you must change `/home/git/gitlab/config/resque.yml`, -following the example in `/home/git/gitlab/config/resque.yml.example` and -uncommenting the sentinels line, changing to the correct server credentials, -and resstart GitLab. - -For a Omnibus install you have to add/change this lines from the -`/etc/gitlab/gitlab.rb` configuration file: - -```ruby -gitlab['gitlab-rails']['redis_host'] = "master-redis.example.com" -gitlab['gitlab-rails']['redis_port'] = 6379 -gitlab['gitlab-rails']['redis_password'] = "redis-secure-password-here" -gitlab['gitlab-rails']['redis_socket'] = nil -gitlab['gitlab-rails']['redis_sentinels'] = [ - {'host' => '10.10.10.1', 'port' => 26379}, - {'host' => '10.10.10.2', 'port' => 26379}, - {'host' => '10.10.10.3', 'port' => 26379} -] -``` - -After the change run the reconfigure command: - -```bash -sudo gitlab-ctl reconfigure - -``` +> **Note:** Redis does not require authentication by default. See + [Redis Security](http://redis.io/topics/security) documentation for more + information. We recommend using a combination of a Redis password and tight + firewall rules to secure your Redis service. ## Configure your own Redis server @@ -116,49 +14,288 @@ If you're hosting GitLab on a cloud provider, you can optionally use a managed service for Redis. For example, AWS offers a managed ElastiCache service that runs Redis. -> **Note:** Redis does not require authentication by default. See - [Redis Security](http://redis.io/topics/security) documentation for more - information. We recommend using a combination of a Redis password and tight - firewall rules to secure your Redis service. +## Configure Redis using Omnibus -## Configure using Omnibus +If you don't want to bother setting up your own Redis server, you can use the +one bundled with Omnibus. In this case, you should disable all services except +Redis. 1. Download/install GitLab Omnibus using **steps 1 and 2** from [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other steps on the download page. 1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. Be sure to change the `external_url` to match your eventual GitLab front-end - URL. + URL: ```ruby - external_url 'https://gitlab.example.com' + external_url 'https://gitlab.example.com' - # Disable all components except Redis - redis['enable'] = true - bootstrap['enable'] = false - nginx['enable'] = false - unicorn['enable'] = false - sidekiq['enable'] = false - postgresql['enable'] = false - gitlab_workhorse['enable'] = false - mailroom['enable'] = false + # Disable all services except Redis + redis['enable'] = true + bootstrap['enable'] = false + nginx['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + postgresql['enable'] = false + gitlab_workhorse['enable'] = false + mailroom['enable'] = false - # Redis configuration - redis['port'] = 6379 - redis['bind'] = '0.0.0.0' + # Redis configuration + redis['port'] = 6379 + redis['bind'] = '0.0.0.0' - # If you wish to use Redis authentication (recommended) - redis['password'] = 'Redis Password' + # If you wish to use Redis authentication (recommended) + redis['password'] = 'Redis Password' ``` 1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL. > **Note**: This `reconfigure` step will result in some errors. That's OK - don't be alarmed. + 1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations from running on upgrade. Only the primary GitLab application server should handle migrations. +## Experimental Redis sentinel support + +> [Introduced][ce-1877] in GitLab 8.10. + +Since GitLab 8.10, you can configure a list of Redis Sentinel servers that +will monitor a group of Redis servers to provide you with a standard failover +support. + +There is currently one exception to the Sentinel support: `mail_room`, the +component that processes incoming emails. It doesn't support Sentinel yet, but +we hope to integrate a future release that does support it. + +To get a better understanding on how to correctly setup Sentinel, please read +the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as +failing to configure it correctly can lead to data loss. + +The configuration consists of three parts: + +- Redis setup +- Sentinel setup +- GitLab setup + +Read carefully how to configure those components below. + +### Sentinel - Redis setup + +You must have at least 2 Redis servers: 1 Master, 1 or more Slaves. +They should be configured the same way and with similar server specs, as +in a failover situation, any Slave can be elected as the new Master by +the Sentinel servers. + +In a minimal setup, the only required change for the slaves in `redis.conf` +is the addition of a `slaveof` line pointing to the initial master. +You can increase the security by defining a `requirepass` configuration in +the master, and `masterauth` in slaves. + +--- + +**Configuring your own Redis server** + +1. Add to the slaves' `redis.conf`: + + ```conf + # IP and port of the master Redis server + slaveof 10.10.10.10 6379 + ``` + +1. Optionally, set up password authentication for increased security. + Add the following to master's `redis.conf`: + + ```conf + # Optional password authentication for increased security + requirepass "" + ``` + +1. Then add this line to all the slave servers' `redis.conf`: + + ```conf + masterauth "" + ``` + +1. Restart the Redis services for the changes to take effect. + +--- + +**Using Redis via Omnibus** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # IP and port of the master Redis server + gitlab['redis']['master_ip'] = '10.10.10.10' + gitlab['redis']['master_port'] = 6379 + ``` + +1. Optionally, set up password authentication for increased security. + Add the following to master's `/etc/gitlab/gitlab.rb`: + + ```conf + # Optional password authentication for increased security + gitlab['redis']['password'] = '' + ``` + +1. Then add this line to all the slave servers' `redis.conf`: + + ```conf + gitlab['redis']['master_password'] = '' + ``` + +1. Reconfigure the GitLab for the changes to take effect. + +--- + +Now that the Redis servers are all set up, let's configure the Sentinel +servers. + +### Sentinel setup + +The support for Sentinel in Ruby has some [caveats](https://github.com/redis/redis-rb/issues/531). +While you can give any name for the `master-group-name` part of the +configuration, as in this example: + +```conf +sentinel monitor +``` + +,for it to work in Ruby, you have to use the "hostname" of the master Redis +server, otherwise you will get an error message like: +`Redis::CannotConnectError: No sentinels available.`. Read +[Sentinel troubleshooting](#sentinel-troubleshooting) for more information. + +Here is an example configuration file (`sentinel.conf`) for a Sentinel node: + +```conf +port 26379 +sentinel monitor master-redis.example.com 10.10.10.10 6379 1 +sentinel down-after-milliseconds master-redis.example.com 10000 +sentinel config-epoch master-redis.example.com 0 +sentinel leader-epoch master-redis.example.com 0 +``` + +--- + +The final part is to inform the main GitLab application server of the Redis +master and the new sentinels servers. + +### Sentinel - GitLab setup + +You can enable or disable sentinel support at any time in new or existing +installations. From the GitLab application perspective, all it requires is +the correct credentials for the Master redis and for a few Sentinels nodes. + +It doesn't require a list of all sentinel nodes, as in case of a failure, +the application will need to query only one of them. + +>**Note:** +The following steps should be performed in the [GitLab application server](gitlab.md). + +**For source based installations** + +1. Edit `/home/git/gitlab/config/resque.yml` following the example in + `/home/git/gitlab/config/resque.yml.example`i, and uncomment the sentinels + line, changing to the correct server credentials. +1. Restart GitLab for the changes to take effect. + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: + + ```ruby + gitlab-rails['redis_host'] = "master-redis.example.com" + gitlab-rails['redis_port'] = 6379 + gitlab-rails['redis_password'] = "redis-secure-password-here" + gitlab-rails['redis_socket'] = nil + gitlab-rails['redis_sentinels'] = [ + {'host' => '10.10.10.1', 'port' => 26379}, + {'host' => '10.10.10.2', 'port' => 26379}, + {'host' => '10.10.10.3', 'port' => 26379} + ] + ``` + +1. [Reconfigure] the GitLab for the changes to take effect. + +### Sentinel troubleshooting + +If you get an error like: `Redis::CannotConnectError: No sentinels available.`, +there may be something wrong with your configuration files or it can be related +to [this issue][gh-531] ([pull request][gh-534] that should make things better). + +It's a bit rigid the way you have to config `resque.yml` and `sentinel.conf`, +otherwise `redis-rb` will not work properly. + +The hostname ('my-primary-redis') of the primary Redis server (`sentinel.conf`) +**must** match the one configured in GitLab (`resque.yml` for source installations +or `gitlab-rails['redis_*']` in Omnibus) and it must be valid ex: + +```conf +# sentinel.conf: +sentinel monitor my-primary-redis 10.10.10.10 6379 1 +sentinel down-after-milliseconds my-primary-redis 10000 +sentinel config-epoch my-primary-redis 0 +sentinel leader-epoch my-primary-redis 0 +``` + +```yaml +# resque.yaml +production: + url: redis://my-primary-redis:6378 + sentinels: + - + host: slave1 + port: 26380 # point to sentinel, not to redis port + - + host: slave2 + port: 26381 # point to sentinel, not to redis port +``` + +When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel) + +--- + +To make sure your configuration is correct: + +1. SSH into your GitLab application server +1. Enter the Rails console: + + ``` + # For Omnibus installations + sudo gitlab-rails console + + # For source installations + sudo -u git rails console RAILS_ENV=production + ``` + +1. Run in the console: + + ```ruby + redis = Redis.new(Gitlab::Redis.params) + redis.info + ``` + + Keep this screen open and try to simulate a failover below. + +1. To simulate a failover on master Redis, SSH into the Redis server and run: + + ```bash + # port must match your master redis port + redis-cli -h localhost -p 6379 DEBUG sleep 60 + ``` + +1. Then back in the Rails console from the first step, run: + + ``` + redis.info + ``` + + You should see a different port after a few seconds delay + (the failover/reconnect time). + --- Read more on high-availability configuration: @@ -167,3 +304,9 @@ Read more on high-availability configuration: 1. [Configure NFS](nfs.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) + +[ce-1877]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1877 +[restart]: ../restart_gitlab.md#installations-from-source +[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure +[gh-531]: https://github.com/redis/redis-rb/issues/531 +[gh-534]: https://github.com/redis/redis-rb/issues/534