Refactor Redis HA docs to also include Sentinel
This commit is contained in:
parent
a5c1e2e55e
commit
ca7c6cc7d6
1 changed files with 272 additions and 129 deletions
|
@ -1,114 +1,12 @@
|
|||
# Configuring Redis for GitLab HA
|
||||
|
||||
You can choose to install and manage Redis yourself, or you can use GitLab
|
||||
Omnibus packages to help.
|
||||
You can choose to install and manage Redis yourself, or you can use the one
|
||||
that comes bundled with GitLab Omnibus packages.
|
||||
|
||||
## Experimental Redis Sentinel support
|
||||
|
||||
Since 8.10 release, you can configure a list of Redis Sentinel servers that
|
||||
will monitor a group of Redis servers to provide you with a standard failover
|
||||
support.
|
||||
|
||||
There is currently one exception to the Sentinel support: **mail_room**, the
|
||||
component that process incoming emails.
|
||||
|
||||
It doesn't support Sentinel yet, but we hope to integrate a future release
|
||||
that does support it.
|
||||
|
||||
To get a better understanding on how to correctly setup Sentinel, please read
|
||||
[Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
|
||||
faling to configure it correctly can lead to data-loss.
|
||||
|
||||
### Redis setup
|
||||
|
||||
You must have at least 2 Redis servers: 1 Master, 1 or more Slaves.
|
||||
They should be configured the same way and with similar server specs, as
|
||||
in a failover situation, any Slave can be elected as the new Master by
|
||||
the Sentinels servers.
|
||||
|
||||
In a minimal setup, the only required change for the slaves in `redis.conf`
|
||||
is the addition of a `slaveof` line pointing to the initial master like this:
|
||||
|
||||
```conf
|
||||
slaveof 192.168.1.1 6379
|
||||
```
|
||||
|
||||
You can increase the security by defining a `requirepass` configuration in
|
||||
the master:
|
||||
|
||||
```conf
|
||||
requirepass "<password>
|
||||
```
|
||||
|
||||
and adding this line to all the slave servers:
|
||||
|
||||
```conf
|
||||
masterauth "<password>"
|
||||
```
|
||||
|
||||
> **Note** This setup is not safe to be used by a machine accessible by the
|
||||
internet. Use it in combination with tight firewall rules.
|
||||
|
||||
### Sentinel setup
|
||||
|
||||
The support for Sentinel in ruby have some [caveats](https://github.com/redis/redis-rb/issues/531).
|
||||
While you can give any name for the `master-group-name` part of the
|
||||
configuration, as in this example:
|
||||
|
||||
```conf
|
||||
sentinel monitor <master-group-name> <ip> <port> <quorum>`
|
||||
```
|
||||
|
||||
For it to work in ruby, you have to use the "hostname" of the master redis
|
||||
server otherwhise you will get an error message like this one:
|
||||
`Redis::CannotConnectError: No sentinels available.`.
|
||||
|
||||
|
||||
Here is an example configuration file (`sentinel.conf`) for a Sentinel node:
|
||||
|
||||
```conf
|
||||
port 26379
|
||||
sentinel monitor master-redis.example.com 10.10.10.10 6379 1
|
||||
sentinel down-after-milliseconds master-redis.example.com 10000
|
||||
sentinel config-epoch master-redis.example.com 0
|
||||
sentinel leader-epoch locmaster-redis.example.comalhost 0
|
||||
```
|
||||
|
||||
### GitLab setup
|
||||
|
||||
You can enable or disable sentinel support at any time in new or existing
|
||||
installs. From the GitLab application perspective, all it requires is
|
||||
the correct credentials for the Master redis and for a few Sentinels nodes.
|
||||
|
||||
It doesn't require a list of all sentinel nodes, as in case of a failure,
|
||||
the application will need to query only one of them.
|
||||
|
||||
For a source based install, you must change `/home/git/gitlab/config/resque.yml`,
|
||||
following the example in `/home/git/gitlab/config/resque.yml.example` and
|
||||
uncommenting the sentinels line, changing to the correct server credentials,
|
||||
and resstart GitLab.
|
||||
|
||||
For a Omnibus install you have to add/change this lines from the
|
||||
`/etc/gitlab/gitlab.rb` configuration file:
|
||||
|
||||
```ruby
|
||||
gitlab['gitlab-rails']['redis_host'] = "master-redis.example.com"
|
||||
gitlab['gitlab-rails']['redis_port'] = 6379
|
||||
gitlab['gitlab-rails']['redis_password'] = "redis-secure-password-here"
|
||||
gitlab['gitlab-rails']['redis_socket'] = nil
|
||||
gitlab['gitlab-rails']['redis_sentinels'] = [
|
||||
{'host' => '10.10.10.1', 'port' => 26379},
|
||||
{'host' => '10.10.10.2', 'port' => 26379},
|
||||
{'host' => '10.10.10.3', 'port' => 26379}
|
||||
]
|
||||
```
|
||||
|
||||
After the change run the reconfigure command:
|
||||
|
||||
```bash
|
||||
sudo gitlab-ctl reconfigure
|
||||
|
||||
```
|
||||
> **Note:** Redis does not require authentication by default. See
|
||||
[Redis Security](http://redis.io/topics/security) documentation for more
|
||||
information. We recommend using a combination of a Redis password and tight
|
||||
firewall rules to secure your Redis service.
|
||||
|
||||
## Configure your own Redis server
|
||||
|
||||
|
@ -116,24 +14,23 @@ If you're hosting GitLab on a cloud provider, you can optionally use a
|
|||
managed service for Redis. For example, AWS offers a managed ElastiCache service
|
||||
that runs Redis.
|
||||
|
||||
> **Note:** Redis does not require authentication by default. See
|
||||
[Redis Security](http://redis.io/topics/security) documentation for more
|
||||
information. We recommend using a combination of a Redis password and tight
|
||||
firewall rules to secure your Redis service.
|
||||
## Configure Redis using Omnibus
|
||||
|
||||
## Configure using Omnibus
|
||||
If you don't want to bother setting up your own Redis server, you can use the
|
||||
one bundled with Omnibus. In this case, you should disable all services except
|
||||
Redis.
|
||||
|
||||
1. Download/install GitLab Omnibus using **steps 1 and 2** from
|
||||
[GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
|
||||
steps on the download page.
|
||||
1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
|
||||
Be sure to change the `external_url` to match your eventual GitLab front-end
|
||||
URL.
|
||||
URL:
|
||||
|
||||
```ruby
|
||||
external_url 'https://gitlab.example.com'
|
||||
|
||||
# Disable all components except Redis
|
||||
# Disable all services except Redis
|
||||
redis['enable'] = true
|
||||
bootstrap['enable'] = false
|
||||
nginx['enable'] = false
|
||||
|
@ -155,10 +52,250 @@ that runs Redis.
|
|||
|
||||
> **Note**: This `reconfigure` step will result in some errors.
|
||||
That's OK - don't be alarmed.
|
||||
|
||||
1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations
|
||||
from running on upgrade. Only the primary GitLab application server should
|
||||
handle migrations.
|
||||
|
||||
## Experimental Redis sentinel support
|
||||
|
||||
> [Introduced][ce-1877] in GitLab 8.10.
|
||||
|
||||
Since GitLab 8.10, you can configure a list of Redis Sentinel servers that
|
||||
will monitor a group of Redis servers to provide you with a standard failover
|
||||
support.
|
||||
|
||||
There is currently one exception to the Sentinel support: `mail_room`, the
|
||||
component that processes incoming emails. It doesn't support Sentinel yet, but
|
||||
we hope to integrate a future release that does support it.
|
||||
|
||||
To get a better understanding on how to correctly setup Sentinel, please read
|
||||
the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
|
||||
failing to configure it correctly can lead to data loss.
|
||||
|
||||
The configuration consists of three parts:
|
||||
|
||||
- Redis setup
|
||||
- Sentinel setup
|
||||
- GitLab setup
|
||||
|
||||
Read carefully how to configure those components below.
|
||||
|
||||
### Sentinel - Redis setup
|
||||
|
||||
You must have at least 2 Redis servers: 1 Master, 1 or more Slaves.
|
||||
They should be configured the same way and with similar server specs, as
|
||||
in a failover situation, any Slave can be elected as the new Master by
|
||||
the Sentinel servers.
|
||||
|
||||
In a minimal setup, the only required change for the slaves in `redis.conf`
|
||||
is the addition of a `slaveof` line pointing to the initial master.
|
||||
You can increase the security by defining a `requirepass` configuration in
|
||||
the master, and `masterauth` in slaves.
|
||||
|
||||
---
|
||||
|
||||
**Configuring your own Redis server**
|
||||
|
||||
1. Add to the slaves' `redis.conf`:
|
||||
|
||||
```conf
|
||||
# IP and port of the master Redis server
|
||||
slaveof 10.10.10.10 6379
|
||||
```
|
||||
|
||||
1. Optionally, set up password authentication for increased security.
|
||||
Add the following to master's `redis.conf`:
|
||||
|
||||
```conf
|
||||
# Optional password authentication for increased security
|
||||
requirepass "<password>"
|
||||
```
|
||||
|
||||
1. Then add this line to all the slave servers' `redis.conf`:
|
||||
|
||||
```conf
|
||||
masterauth "<password>"
|
||||
```
|
||||
|
||||
1. Restart the Redis services for the changes to take effect.
|
||||
|
||||
---
|
||||
|
||||
**Using Redis via Omnibus**
|
||||
|
||||
1. Edit `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```ruby
|
||||
# IP and port of the master Redis server
|
||||
gitlab['redis']['master_ip'] = '10.10.10.10'
|
||||
gitlab['redis']['master_port'] = 6379
|
||||
```
|
||||
|
||||
1. Optionally, set up password authentication for increased security.
|
||||
Add the following to master's `/etc/gitlab/gitlab.rb`:
|
||||
|
||||
```conf
|
||||
# Optional password authentication for increased security
|
||||
gitlab['redis']['password'] = '<password>'
|
||||
```
|
||||
|
||||
1. Then add this line to all the slave servers' `redis.conf`:
|
||||
|
||||
```conf
|
||||
gitlab['redis']['master_password'] = '<password>'
|
||||
```
|
||||
|
||||
1. Reconfigure the GitLab for the changes to take effect.
|
||||
|
||||
---
|
||||
|
||||
Now that the Redis servers are all set up, let's configure the Sentinel
|
||||
servers.
|
||||
|
||||
### Sentinel setup
|
||||
|
||||
The support for Sentinel in Ruby has some [caveats](https://github.com/redis/redis-rb/issues/531).
|
||||
While you can give any name for the `master-group-name` part of the
|
||||
configuration, as in this example:
|
||||
|
||||
```conf
|
||||
sentinel monitor <master-group-name> <ip> <port> <quorum>
|
||||
```
|
||||
|
||||
,for it to work in Ruby, you have to use the "hostname" of the master Redis
|
||||
server, otherwise you will get an error message like:
|
||||
`Redis::CannotConnectError: No sentinels available.`. Read
|
||||
[Sentinel troubleshooting](#sentinel-troubleshooting) for more information.
|
||||
|
||||
Here is an example configuration file (`sentinel.conf`) for a Sentinel node:
|
||||
|
||||
```conf
|
||||
port 26379
|
||||
sentinel monitor master-redis.example.com 10.10.10.10 6379 1
|
||||
sentinel down-after-milliseconds master-redis.example.com 10000
|
||||
sentinel config-epoch master-redis.example.com 0
|
||||
sentinel leader-epoch master-redis.example.com 0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
The final part is to inform the main GitLab application server of the Redis
|
||||
master and the new sentinels servers.
|
||||
|
||||
### Sentinel - GitLab setup
|
||||
|
||||
You can enable or disable sentinel support at any time in new or existing
|
||||
installations. From the GitLab application perspective, all it requires is
|
||||
the correct credentials for the Master redis and for a few Sentinels nodes.
|
||||
|
||||
It doesn't require a list of all sentinel nodes, as in case of a failure,
|
||||
the application will need to query only one of them.
|
||||
|
||||
>**Note:**
|
||||
The following steps should be performed in the [GitLab application server](gitlab.md).
|
||||
|
||||
**For source based installations**
|
||||
|
||||
1. Edit `/home/git/gitlab/config/resque.yml` following the example in
|
||||
`/home/git/gitlab/config/resque.yml.example`i, and uncomment the sentinels
|
||||
line, changing to the correct server credentials.
|
||||
1. Restart GitLab for the changes to take effect.
|
||||
|
||||
**For Omnibus installations**
|
||||
|
||||
1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:
|
||||
|
||||
```ruby
|
||||
gitlab-rails['redis_host'] = "master-redis.example.com"
|
||||
gitlab-rails['redis_port'] = 6379
|
||||
gitlab-rails['redis_password'] = "redis-secure-password-here"
|
||||
gitlab-rails['redis_socket'] = nil
|
||||
gitlab-rails['redis_sentinels'] = [
|
||||
{'host' => '10.10.10.1', 'port' => 26379},
|
||||
{'host' => '10.10.10.2', 'port' => 26379},
|
||||
{'host' => '10.10.10.3', 'port' => 26379}
|
||||
]
|
||||
```
|
||||
|
||||
1. [Reconfigure] the GitLab for the changes to take effect.
|
||||
|
||||
### Sentinel troubleshooting
|
||||
|
||||
If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
|
||||
there may be something wrong with your configuration files or it can be related
|
||||
to [this issue][gh-531] ([pull request][gh-534] that should make things better).
|
||||
|
||||
It's a bit rigid the way you have to config `resque.yml` and `sentinel.conf`,
|
||||
otherwise `redis-rb` will not work properly.
|
||||
|
||||
The hostname ('my-primary-redis') of the primary Redis server (`sentinel.conf`)
|
||||
**must** match the one configured in GitLab (`resque.yml` for source installations
|
||||
or `gitlab-rails['redis_*']` in Omnibus) and it must be valid ex:
|
||||
|
||||
```conf
|
||||
# sentinel.conf:
|
||||
sentinel monitor my-primary-redis 10.10.10.10 6379 1
|
||||
sentinel down-after-milliseconds my-primary-redis 10000
|
||||
sentinel config-epoch my-primary-redis 0
|
||||
sentinel leader-epoch my-primary-redis 0
|
||||
```
|
||||
|
||||
```yaml
|
||||
# resque.yaml
|
||||
production:
|
||||
url: redis://my-primary-redis:6378
|
||||
sentinels:
|
||||
-
|
||||
host: slave1
|
||||
port: 26380 # point to sentinel, not to redis port
|
||||
-
|
||||
host: slave2
|
||||
port: 26381 # point to sentinel, not to redis port
|
||||
```
|
||||
|
||||
When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel)
|
||||
|
||||
---
|
||||
|
||||
To make sure your configuration is correct:
|
||||
|
||||
1. SSH into your GitLab application server
|
||||
1. Enter the Rails console:
|
||||
|
||||
```
|
||||
# For Omnibus installations
|
||||
sudo gitlab-rails console
|
||||
|
||||
# For source installations
|
||||
sudo -u git rails console RAILS_ENV=production
|
||||
```
|
||||
|
||||
1. Run in the console:
|
||||
|
||||
```ruby
|
||||
redis = Redis.new(Gitlab::Redis.params)
|
||||
redis.info
|
||||
```
|
||||
|
||||
Keep this screen open and try to simulate a failover below.
|
||||
|
||||
1. To simulate a failover on master Redis, SSH into the Redis server and run:
|
||||
|
||||
```bash
|
||||
# port must match your master redis port
|
||||
redis-cli -h localhost -p 6379 DEBUG sleep 60
|
||||
```
|
||||
|
||||
1. Then back in the Rails console from the first step, run:
|
||||
|
||||
```
|
||||
redis.info
|
||||
```
|
||||
|
||||
You should see a different port after a few seconds delay
|
||||
(the failover/reconnect time).
|
||||
|
||||
---
|
||||
|
||||
Read more on high-availability configuration:
|
||||
|
@ -167,3 +304,9 @@ Read more on high-availability configuration:
|
|||
1. [Configure NFS](nfs.md)
|
||||
1. [Configure the GitLab application servers](gitlab.md)
|
||||
1. [Configure the load balancers](load_balancer.md)
|
||||
|
||||
[ce-1877]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1877
|
||||
[restart]: ../restart_gitlab.md#installations-from-source
|
||||
[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure
|
||||
[gh-531]: https://github.com/redis/redis-rb/issues/531
|
||||
[gh-534]: https://github.com/redis/redis-rb/issues/534
|
||||
|
|
Loading…
Reference in a new issue