kotovalexarian-likes-gitlab/gitlab-org--gitlab-foss

Gabriel Mazetto c76ff8cf32 Improved redis sentinel documentation for CE

2016-11-08 06:38:29 +01:00

9.2 KiB

Raw Blame History

Configuring Redis for GitLab HA

You can choose to install and manage Redis yourself, or you can use the one that comes bundled with GitLab Omnibus packages.

Note: Redis does not require authentication by default. See Redis Security documentation for more information. We recommend using a combination of a Redis password and tight firewall rules to secure your Redis service.

Configure your own Redis server

If you're hosting GitLab on a cloud provider, you can optionally use a managed service for Redis. For example, AWS offers a managed ElastiCache service that runs Redis.

Configure Redis using Omnibus

If you don't want to bother setting up your own Redis server, you can use the one bundled with Omnibus. In this case, you should disable all services except Redis.

Download/install GitLab Omnibus using steps 1 and 2 from GitLab downloads. Do not complete other steps on the download page.

Create/edit /etc/gitlab/gitlab.rb and use the following configuration. Be sure to change the external_url to match your eventual GitLab front-end URL:

external_url 'https://gitlab.example.com'

# Disable all services except Redis
redis['enable'] = true
bootstrap['enable'] = false
nginx['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
postgresql['enable'] = false
gitlab_workhorse['enable'] = false
mailroom['enable'] = false

# Redis configuration
redis['port'] = 6379
redis['bind'] = '0.0.0.0'

# If you wish to use Redis authentication (recommended)
redis['password'] = 'Redis Password'

Run sudo gitlab-ctl reconfigure to install and configure PostgreSQL.

Note

: This reconfigure step will result in some errors. That's OK - don't be alarmed.
Run touch /etc/gitlab/skip-auto-migrations to prevent database migrations from running on upgrade. Only the primary GitLab application server should handle migrations.

Experimental Redis Sentinel support

Introduced in GitLab 8.11.

Since GitLab 8.11, you can configure a list of Redis Sentinel servers that will monitor a group of Redis servers to provide you with a standard failover support.

There is currently one exception to the Sentinel support: mail_room, the component that processes incoming emails. It doesn't support Sentinel yet, but we hope to integrate a future release that does support it.

To get a better understanding on how to correctly setup Sentinel, please read the Redis Sentinel documentation first, as failing to configure it correctly can lead to data loss.

The configuration consists of three parts:

Redis setup
Sentinel setup
GitLab setup

Read carefully how to configure those components below.

Redis setup

You must have at least 2 Redis servers: 1 Master, 1 or more Slaves. They should be configured the same way and with similar server specs, as in a failover situation, any Slave can be elected as the new Master by the Sentinel servers.

In a minimal setup, the only required change for the slaves in redis.conf is the addition of a slaveof line pointing to the initial master. You can increase the security by defining a requirepass configuration in the master, and masterauth in slaves.

Configuring your own Redis server

Add to the slaves' redis.conf:

# IP and port of the master Redis server
slaveof 10.10.10.10 6379

Optionally, set up password authentication for increased security. Add the following to master's redis.conf:
```
# Optional password authentication for increased security
requirepass "<password>"
```
Then add this line to all the slave servers' redis.conf:
```
masterauth "<password>"
```
Restart the Redis services for the changes to take effect.

Using Redis via Omnibus

Edit /etc/gitlab/gitlab.rb of a master Redis machine (usualy a single machine):

## Redis TCP support (will disable UNIX socket transport)
redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
redis['port'] = 6379

## Master redis instance
redis['password'] = '<huge password string here>'

Edit /etc/gitlab/gitlab.rb of a slave Redis machine (should be one or more machines):

## Redis TCP support (will disable UNIX socket transport)
redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
redis['port'] = 6379

## Slave redis instance
redis['master'] = false
redis['master_ip'] = '10.10.10.10' # IP of master Redis server
redis['master_port'] = 6379 # Port of master Redis server
redis['master_password'] = "<huge password string here>"

Reconfigure the GitLab for the changes to take effect: sudo gitlab-ctl reconfigure

Now that the Redis servers are all set up, let's configure the Sentinel servers.

Sentinel setup

We provide an automated way to setup and run the Sentinel daemon with GitLab EE.

See the instructions below how to setup it by yourself.

Here is an example configuration file (sentinel.conf) for a Sentinel node:

port 26379
sentinel monitor gitlab-redis 10.0.0.1 6379 1
sentinel down-after-milliseconds gitlab-redis 10000
sentinel config-epoch gitlab-redis 0
sentinel leader-epoch gitlab-redis 0

The final part is to inform the main GitLab application server of the Redis master and the new sentinels servers.

GitLab setup

You can enable or disable sentinel support at any time in new or existing installations. From the GitLab application perspective, all it requires is the correct credentials for the master Redis and for a few Sentinel nodes.

It doesn't require a list of all Sentinel nodes, as in case of a failure, the application will need to query only one of them.

Note: The following steps should be performed in the GitLab application server.

For source based installations

Edit /home/git/gitlab/config/resque.yml following the example in /home/git/gitlab/config/resque.yml.example, and uncomment the sentinels line, changing to the correct server credentials.
Restart GitLab for the changes to take effect.

For Omnibus installations

Edit /etc/gitlab/gitlab.rb and add/change the following lines:

redis['master_name'] = "gitlab-redis"
redis['master_ip'] = "10.0.0.1"
redis['master_port'] = 6379
redis['master_password'] = '<huge password string here>'
gitlab_rails['redis_sentinels'] = [
  {'host' => '10.10.10.1', 'port' => 26379},
  {'host' => '10.10.10.2', 'port' => 26379},
  {'host' => '10.10.10.3', 'port' => 26379}
]

Reconfigure the GitLab for the changes to take effect.

Sentinel troubleshooting

If you get an error like: Redis::CannotConnectError: No sentinels available., there may be something wrong with your configuration files or it can be related to this issue.

It's a bit non-intuitive the way you have to config resque.yml and sentinel.conf, otherwise redis-rb will not work properly.

The master-group-name ('gitlab-redis') defined in (sentinel.conf) must be used as the hostname in GitLab (resque.yml for source installations or gitlab-rails['redis_*'] in Omnibus):

# sentinel.conf:
sentinel monitor gitlab-redis 10.10.10.10 6379 1
sentinel down-after-milliseconds gitlab-redis 10000
sentinel config-epoch gitlab-redis 0
sentinel leader-epoch gitlab-redis 0

# resque.yaml
production:
  url: redis://:myredispassword@gitlab-redis/
  sentinels:
    -
      host: slave1.example.com # or use ip
      port: 26380 # point to sentinel, not to redis port
    -
      host: slave2.exampl.com # or use ip
      port: 26381 # point to sentinel, not to redis port

When in doubt, please read Redis Sentinel documentation

To make sure your configuration is correct:

SSH into your GitLab application server

Enter the Rails console:

# For Omnibus installations
sudo gitlab-rails console

# For source installations
sudo -u git rails console RAILS_ENV=production

Run in the console:
```
redis = Redis.new(Gitlab::Redis.params)
redis.info
```
Keep this screen open and try to simulate a failover below.

To simulate a failover on master Redis, SSH into the Redis server and run:

# port must match your master redis port
 redis-cli -h localhost -p 6379 DEBUG sleep 60

Then back in the Rails console from the first step, run:
```
redis.info
```
You should see a different port after a few seconds delay (the failover/reconnect time).

Read more on high-availability configuration:

9.2 KiB Raw Blame History