8.7 KiB
Redis guidelines
GitLab uses Redis for the following distinct purposes:
- Caching (mostly via
Rails.cache
). - As a job processing queue with Sidekiq.
- To manage the shared application state.
- As a Pub/Sub queue backend for ActionCable.
In most environments (including the GDK), all of these point to the same Redis instance.
On GitLab.com, we use separate Redis instances. (We do not currently use ActionCable on GitLab.com).
Every application process is configured to use the same Redis servers, so they can be used for inter-process communication in cases where PostgreSQL is less appropriate. For example, transient state or data that is written much more often than it is read.
If Geo is enabled, each Geo node gets its own, independent Redis database.
Key naming
Redis is a flat namespace with no hierarchy, which means we must pay attention
to key names to avoid collisions. Typically we use colon-separated elements to
provide a semblance of structure at application level. An example might be
projects:1:somekey
.
Although we split our Redis usage by purpose into distinct categories, and those may map to separate Redis servers in a Highly Available configuration like GitLab.com, the default Omnibus and GDK setups share a single Redis server. This means that keys should always be globally unique across all categories.
It is usually better to use immutable identifiers - project ID rather than full path, for instance - in Redis key names. If full path is used, the key will stop being consulted if the project is renamed. If the contents of the key are invalidated by a name change, it is better to include a hook that will expire the entry, instead of relying on the key changing.
Multi-key commands
We don't use Redis Cluster at the moment, but may wish to in the future: #118820.
This imposes an additional constraint on naming: where GitLab is performing operations that require several keys to be held on the same Redis server - for instance, diffing two sets held in Redis - the keys should ensure that by enclosing the changeable parts in curly braces. For example:
project:{1}:set_a
project:{1}:set_b
project:{2}:set_c
set_a
and set_b
are guaranteed to be held on the same Redis server, while set_c
is not.
Currently, we validate this in the development and test environments
with the RedisClusterValidator
,
which is enabled for the cache
and shared_state
Redis instances..
Redis in structured logging
For GitLab Team Members: There are basic and advanced videos that show how you can work with the Redis structured logging fields on GitLab.com.
Our structured logging for web requests and Sidekiq jobs contains fields for the duration, call count, bytes written, and bytes read per Redis instance, along with a total for all Redis instances. For a particular request, this might look like:
Field | Value |
---|---|
json.queue_duration_s |
0.01 |
json.redis_cache_calls |
1 |
json.redis_cache_duration_s |
0 |
json.redis_cache_read_bytes |
109 |
json.redis_cache_write_bytes |
49 |
json.redis_calls |
2 |
json.redis_duration_s |
0.001 |
json.redis_read_bytes |
111 |
json.redis_shared_state_calls |
1 |
json.redis_shared_state_duration_s |
0 |
json.redis_shared_state_read_bytes |
2 |
json.redis_shared_state_write_bytes |
206 |
json.redis_write_bytes |
255 |
As all of these fields are indexed, it is then straightforward to
investigate Redis usage in production. For instance, to find the
requests that read the most data from the cache, we can just sort by
redis_cache_read_bytes
in descending order.
The slow log
TIP: Tip: There is a video showing how to see the slow log (GitLab internal) on GitLab.com
On GitLab.com, entries from the Redis
slow log are available in the
pubsub-redis-inf-gprd*
index with the redis.slowlog
tag.
This shows commands that have taken a long time and may be a performance
concern.
The fluent-plugin-redis-slowlog project is responsible for taking the slowlog entries from Redis and passing to fluentd (and ultimately Elasticsearch).
Analyzing the entire keyspace
The Redis Keyspace Analyzer project contains tools for dumping the full key list and memory usage of a Redis instance, and then analyzing those lists while elimating potentially sensitive data from the results. It can be used to find the most frequent key patterns, or those that use the most memory.
Currently this is not run automatically for the GitLab.com Redis instances, but is run manually on an as-needed basis.
Utility classes
We have some extra classes to help with specific use cases. These are
mostly for fine-grained control of Redis usage, so they wouldn't be used
in combination with the Rails.cache
wrapper: we'd either use
Rails.cache
or these classes and literal Redis commands.
Rails.cache
or these classes and literal Redis commands. We prefer
using Rails.cache
so we can reap the benefits of future optimizations
done to Rails. It is worth noting that Ruby objects are
marshalled
when written to Redis, so we need to pay attention to not to store huge
objects, or untrusted user input.
Typically we would only use these classes when at least one of the following is true:
- We want to manipulate data on a non-cache Redis instance.
Rails.cache
does not support the operations we want to perform.
Gitlab::Redis::{Cache,SharedState,Queues}
These classes wrap the Redis instances (using
Gitlab::Redis::Wrapper
)
to make it convenient to work with them directly. The typical use is to
call .with
on the class, which takes a block that yields the Redis
connection. For example:
# Get the value of `key` from the shared state (persistent) Redis
Gitlab::Redis::SharedState.with { |redis| redis.get(key) }
# Check if `value` is a member of the set `key`
Gitlab::Redis::Cache.with { |redis| redis.sismember(key, value) }
Gitlab::Redis::Boolean
In Redis, every value is a string.
Gitlab::Redis::Boolean
makes sure that booleans are encoded and decoded consistently.
Gitlab::Redis::HLL
The Redis PFCOUNT
,
PFADD
, and
PFMERGE
commands operate on
HyperLogLogs, a data structure that allows estimating the number of unique
elements with low memory usage. (In addition to the PFCOUNT
documentation,
Thoughtbot's article on HyperLogLogs in
Redis provides a good
background here.)
Gitlab::Redis::HLL
provides a convenient interface for adding and counting values in HyperLogLogs.
Gitlab::SetCache
For cases where we need to efficiently check the whether an item is in a group
of items, we can use a Redis set.
Gitlab::SetCache
provides an #include?
method that will use the
SISMEMBER
command, as well as #read
to fetch all entries in the set.
This is used by the
RepositorySetCache
to provide a convenient way to use sets to cache repository data like branch
names.