2016-04-22 18:18:06 +00:00
|
|
|
# Configuring Redis for GitLab HA
|
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
You can choose to install and manage Redis yourself, or you can use the one
|
|
|
|
that comes bundled with GitLab Omnibus packages.
|
2016-04-22 18:18:06 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
> **Note:** Redis does not require authentication by default. See
|
|
|
|
[Redis Security](http://redis.io/topics/security) documentation for more
|
|
|
|
information. We recommend using a combination of a Redis password and tight
|
|
|
|
firewall rules to secure your Redis service.
|
|
|
|
|
|
|
|
## Configure your own Redis server
|
|
|
|
|
|
|
|
If you're hosting GitLab on a cloud provider, you can optionally use a
|
|
|
|
managed service for Redis. For example, AWS offers a managed ElastiCache service
|
|
|
|
that runs Redis.
|
|
|
|
|
|
|
|
## Configure Redis using Omnibus
|
|
|
|
|
|
|
|
If you don't want to bother setting up your own Redis server, you can use the
|
|
|
|
one bundled with Omnibus. In this case, you should disable all services except
|
|
|
|
Redis.
|
|
|
|
|
|
|
|
1. Download/install GitLab Omnibus using **steps 1 and 2** from
|
|
|
|
[GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
|
|
|
|
steps on the download page.
|
|
|
|
1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
|
|
|
|
Be sure to change the `external_url` to match your eventual GitLab front-end
|
|
|
|
URL:
|
|
|
|
|
|
|
|
```ruby
|
|
|
|
external_url 'https://gitlab.example.com'
|
|
|
|
|
|
|
|
# Disable all services except Redis
|
|
|
|
redis['enable'] = true
|
|
|
|
bootstrap['enable'] = false
|
|
|
|
nginx['enable'] = false
|
|
|
|
unicorn['enable'] = false
|
|
|
|
sidekiq['enable'] = false
|
|
|
|
postgresql['enable'] = false
|
|
|
|
gitlab_workhorse['enable'] = false
|
|
|
|
mailroom['enable'] = false
|
|
|
|
|
|
|
|
# Redis configuration
|
|
|
|
redis['port'] = 6379
|
|
|
|
redis['bind'] = '0.0.0.0'
|
|
|
|
|
|
|
|
# If you wish to use Redis authentication (recommended)
|
2016-10-06 23:47:22 +00:00
|
|
|
redis['password'] = 'redis-password-goes-here'
|
2016-07-19 11:27:50 +00:00
|
|
|
```
|
|
|
|
|
|
|
|
1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL.
|
|
|
|
|
|
|
|
> **Note**: This `reconfigure` step will result in some errors.
|
|
|
|
That's OK - don't be alarmed.
|
|
|
|
|
|
|
|
1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations
|
|
|
|
from running on upgrade. Only the primary GitLab application server should
|
|
|
|
handle migrations.
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-08-02 01:25:28 +00:00
|
|
|
## Experimental Redis Sentinel support
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-08-04 16:55:24 +00:00
|
|
|
> [Introduced][ce-1877] in GitLab 8.11.
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-08-04 16:55:24 +00:00
|
|
|
Since GitLab 8.11, you can configure a list of Redis Sentinel servers that
|
2016-07-14 01:57:47 +00:00
|
|
|
will monitor a group of Redis servers to provide you with a standard failover
|
|
|
|
support.
|
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
There is currently one exception to the Sentinel support: `mail_room`, the
|
|
|
|
component that processes incoming emails. It doesn't support Sentinel yet, but
|
|
|
|
we hope to integrate a future release that does support it.
|
2016-07-14 01:57:47 +00:00
|
|
|
|
|
|
|
To get a better understanding on how to correctly setup Sentinel, please read
|
2016-07-19 11:27:50 +00:00
|
|
|
the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
|
|
|
|
failing to configure it correctly can lead to data loss.
|
|
|
|
|
|
|
|
The configuration consists of three parts:
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
- Redis setup
|
|
|
|
- Sentinel setup
|
|
|
|
- GitLab setup
|
|
|
|
|
|
|
|
Read carefully how to configure those components below.
|
|
|
|
|
2016-08-02 01:25:28 +00:00
|
|
|
### Redis setup
|
2016-07-14 01:57:47 +00:00
|
|
|
|
|
|
|
You must have at least 2 Redis servers: 1 Master, 1 or more Slaves.
|
|
|
|
They should be configured the same way and with similar server specs, as
|
|
|
|
in a failover situation, any Slave can be elected as the new Master by
|
2016-07-19 11:27:50 +00:00
|
|
|
the Sentinel servers.
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
In a minimal setup, the only required change for the slaves in `redis.conf`
|
|
|
|
is the addition of a `slaveof` line pointing to the initial master.
|
|
|
|
You can increase the security by defining a `requirepass` configuration in
|
|
|
|
the master, and `masterauth` in slaves.
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
---
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
**Configuring your own Redis server**
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
1. Add to the slaves' `redis.conf`:
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
```conf
|
|
|
|
# IP and port of the master Redis server
|
|
|
|
slaveof 10.10.10.10 6379
|
|
|
|
```
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
1. Optionally, set up password authentication for increased security.
|
|
|
|
Add the following to master's `redis.conf`:
|
|
|
|
|
|
|
|
```conf
|
|
|
|
# Optional password authentication for increased security
|
|
|
|
requirepass "<password>"
|
|
|
|
```
|
|
|
|
|
|
|
|
1. Then add this line to all the slave servers' `redis.conf`:
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
```conf
|
|
|
|
masterauth "<password>"
|
|
|
|
```
|
|
|
|
|
|
|
|
1. Restart the Redis services for the changes to take effect.
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
**Using Redis via Omnibus**
|
|
|
|
|
2016-08-02 01:25:28 +00:00
|
|
|
1. Edit `/etc/gitlab/gitlab.rb` of a master Redis machine (usualy a single machine):
|
2016-07-19 11:27:50 +00:00
|
|
|
|
|
|
|
```ruby
|
2016-08-02 01:25:28 +00:00
|
|
|
## Redis TCP support (will disable UNIX socket transport)
|
|
|
|
redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
|
|
|
|
redis['port'] = 6379
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-08-02 01:25:28 +00:00
|
|
|
## Master redis instance
|
2016-10-06 23:47:22 +00:00
|
|
|
redis['password'] = 'redis-password-goes-here'
|
2016-07-19 11:27:50 +00:00
|
|
|
```
|
|
|
|
|
2016-08-02 01:25:28 +00:00
|
|
|
1. Edit `/etc/gitlab/gitlab.rb` of a slave Redis machine (should be one or more machines):
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-08-02 01:25:28 +00:00
|
|
|
```ruby
|
|
|
|
## Redis TCP support (will disable UNIX socket transport)
|
|
|
|
redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
|
|
|
|
redis['port'] = 6379
|
|
|
|
|
|
|
|
## Slave redis instance
|
2016-09-22 09:05:28 +00:00
|
|
|
redis['master'] = false
|
2016-08-02 01:25:28 +00:00
|
|
|
redis['master_ip'] = '10.10.10.10' # IP of master Redis server
|
|
|
|
redis['master_port'] = 6379 # Port of master Redis server
|
2016-10-06 23:47:22 +00:00
|
|
|
redis['master_password'] = "redis-password-goes-here"
|
2016-07-19 11:27:50 +00:00
|
|
|
```
|
|
|
|
|
2016-08-02 01:25:28 +00:00
|
|
|
1. Reconfigure the GitLab for the changes to take effect: `sudo gitlab-ctl reconfigure`
|
2016-07-19 11:27:50 +00:00
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
Now that the Redis servers are all set up, let's configure the Sentinel
|
|
|
|
servers.
|
2016-07-14 01:57:47 +00:00
|
|
|
|
|
|
|
### Sentinel setup
|
|
|
|
|
2016-09-22 09:05:28 +00:00
|
|
|
We provide an automated way to setup and run the Sentinel daemon
|
|
|
|
with GitLab EE.
|
2016-08-02 01:25:28 +00:00
|
|
|
|
2016-09-22 09:05:28 +00:00
|
|
|
See the instructions below how to setup it by yourself.
|
2016-07-14 01:57:47 +00:00
|
|
|
|
|
|
|
Here is an example configuration file (`sentinel.conf`) for a Sentinel node:
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-07-14 01:57:47 +00:00
|
|
|
```conf
|
|
|
|
port 26379
|
2016-09-22 09:05:28 +00:00
|
|
|
sentinel monitor gitlab-redis 10.0.0.1 6379 1
|
|
|
|
sentinel down-after-milliseconds gitlab-redis 10000
|
|
|
|
sentinel config-epoch gitlab-redis 0
|
|
|
|
sentinel leader-epoch gitlab-redis 0
|
2016-07-14 01:57:47 +00:00
|
|
|
```
|
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
---
|
|
|
|
|
|
|
|
The final part is to inform the main GitLab application server of the Redis
|
|
|
|
master and the new sentinels servers.
|
|
|
|
|
2016-08-02 01:25:28 +00:00
|
|
|
### GitLab setup
|
2016-07-14 01:57:47 +00:00
|
|
|
|
|
|
|
You can enable or disable sentinel support at any time in new or existing
|
2016-07-19 11:27:50 +00:00
|
|
|
installations. From the GitLab application perspective, all it requires is
|
2016-08-02 01:25:28 +00:00
|
|
|
the correct credentials for the master Redis and for a few Sentinel nodes.
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-08-02 01:25:28 +00:00
|
|
|
It doesn't require a list of all Sentinel nodes, as in case of a failure,
|
2016-07-14 01:57:47 +00:00
|
|
|
the application will need to query only one of them.
|
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
>**Note:**
|
|
|
|
The following steps should be performed in the [GitLab application server](gitlab.md).
|
|
|
|
|
|
|
|
**For source based installations**
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
1. Edit `/home/git/gitlab/config/resque.yml` following the example in
|
2016-08-02 01:25:28 +00:00
|
|
|
`/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels
|
2016-07-19 11:27:50 +00:00
|
|
|
line, changing to the correct server credentials.
|
|
|
|
1. Restart GitLab for the changes to take effect.
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
**For Omnibus installations**
|
2016-07-14 01:57:47 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:
|
|
|
|
|
|
|
|
```ruby
|
2016-09-22 09:05:28 +00:00
|
|
|
redis['master_name'] = "gitlab-redis"
|
2016-10-06 23:47:22 +00:00
|
|
|
redis['master_password'] = 'redis-password-goes-here'
|
2016-09-22 09:05:28 +00:00
|
|
|
gitlab_rails['redis_sentinels'] = [
|
2016-07-19 11:27:50 +00:00
|
|
|
{'host' => '10.10.10.1', 'port' => 26379},
|
|
|
|
{'host' => '10.10.10.2', 'port' => 26379},
|
|
|
|
{'host' => '10.10.10.3', 'port' => 26379}
|
|
|
|
]
|
|
|
|
```
|
|
|
|
|
|
|
|
1. [Reconfigure] the GitLab for the changes to take effect.
|
|
|
|
|
|
|
|
### Sentinel troubleshooting
|
|
|
|
|
2016-10-06 23:47:22 +00:00
|
|
|
#### Omnibus install
|
|
|
|
|
|
|
|
If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
|
|
|
|
there may be something wrong with your configuration files or it can be related
|
|
|
|
to [this issue][gh-531].
|
|
|
|
|
|
|
|
You must make sure you are defining the same value in `redis['master_name']`
|
|
|
|
and `redis['master_pasword']` as you defined for your sentinel node.
|
|
|
|
|
|
|
|
The way the redis connector `redis-rb` works with sentinel is a bit
|
|
|
|
non-intuitive. We try to hide the complexity in omnibus, but it still requires
|
|
|
|
a few extra configs.
|
|
|
|
|
|
|
|
#### Source install
|
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
|
|
|
|
there may be something wrong with your configuration files or it can be related
|
2016-09-22 09:05:28 +00:00
|
|
|
to [this issue][gh-531].
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-09-22 09:05:28 +00:00
|
|
|
It's a bit non-intuitive the way you have to config `resque.yml` and
|
|
|
|
`sentinel.conf`, otherwise `redis-rb` will not work properly.
|
2016-07-19 11:27:50 +00:00
|
|
|
|
2016-09-22 09:05:28 +00:00
|
|
|
The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`)
|
|
|
|
**must** be used as the hostname in GitLab (`resque.yml` for source installations
|
|
|
|
or `gitlab-rails['redis_*']` in Omnibus):
|
2016-07-19 11:27:50 +00:00
|
|
|
|
|
|
|
```conf
|
|
|
|
# sentinel.conf:
|
2016-09-22 09:05:28 +00:00
|
|
|
sentinel monitor gitlab-redis 10.10.10.10 6379 1
|
|
|
|
sentinel down-after-milliseconds gitlab-redis 10000
|
|
|
|
sentinel config-epoch gitlab-redis 0
|
|
|
|
sentinel leader-epoch gitlab-redis 0
|
2016-07-14 01:57:47 +00:00
|
|
|
```
|
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
```yaml
|
|
|
|
# resque.yaml
|
|
|
|
production:
|
2016-09-22 09:05:28 +00:00
|
|
|
url: redis://:myredispassword@gitlab-redis/
|
2016-07-19 11:27:50 +00:00
|
|
|
sentinels:
|
|
|
|
-
|
2016-09-22 09:05:28 +00:00
|
|
|
host: slave1.example.com # or use ip
|
2016-07-19 11:27:50 +00:00
|
|
|
port: 26380 # point to sentinel, not to redis port
|
|
|
|
-
|
2016-09-22 09:05:28 +00:00
|
|
|
host: slave2.exampl.com # or use ip
|
2016-07-19 11:27:50 +00:00
|
|
|
port: 26381 # point to sentinel, not to redis port
|
|
|
|
```
|
2016-04-22 18:18:06 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel)
|
2016-04-22 18:18:06 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
---
|
2016-04-22 18:18:06 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
To make sure your configuration is correct:
|
2016-04-22 18:18:06 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
1. SSH into your GitLab application server
|
|
|
|
1. Enter the Rails console:
|
|
|
|
|
|
|
|
```
|
|
|
|
# For Omnibus installations
|
|
|
|
sudo gitlab-rails console
|
|
|
|
|
|
|
|
# For source installations
|
|
|
|
sudo -u git rails console RAILS_ENV=production
|
|
|
|
```
|
|
|
|
|
|
|
|
1. Run in the console:
|
2016-04-22 18:18:06 +00:00
|
|
|
|
|
|
|
```ruby
|
2016-07-19 11:27:50 +00:00
|
|
|
redis = Redis.new(Gitlab::Redis.params)
|
|
|
|
redis.info
|
2016-04-22 18:18:06 +00:00
|
|
|
```
|
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
Keep this screen open and try to simulate a failover below.
|
2016-04-22 18:18:06 +00:00
|
|
|
|
2016-07-19 11:27:50 +00:00
|
|
|
1. To simulate a failover on master Redis, SSH into the Redis server and run:
|
|
|
|
|
|
|
|
```bash
|
|
|
|
# port must match your master redis port
|
|
|
|
redis-cli -h localhost -p 6379 DEBUG sleep 60
|
|
|
|
```
|
|
|
|
|
|
|
|
1. Then back in the Rails console from the first step, run:
|
|
|
|
|
|
|
|
```
|
|
|
|
redis.info
|
|
|
|
```
|
|
|
|
|
|
|
|
You should see a different port after a few seconds delay
|
|
|
|
(the failover/reconnect time).
|
2016-04-22 18:18:06 +00:00
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
Read more on high-availability configuration:
|
|
|
|
|
|
|
|
1. [Configure the database](database.md)
|
|
|
|
1. [Configure NFS](nfs.md)
|
|
|
|
1. [Configure the GitLab application servers](gitlab.md)
|
|
|
|
1. [Configure the load balancers](load_balancer.md)
|
2016-07-19 11:27:50 +00:00
|
|
|
|
|
|
|
[ce-1877]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1877
|
|
|
|
[restart]: ../restart_gitlab.md#installations-from-source
|
|
|
|
[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure
|
|
|
|
[gh-531]: https://github.com/redis/redis-rb/issues/531
|
|
|
|
[gh-534]: https://github.com/redis/redis-rb/issues/534
|