Merge branch 'docs/refactor-operations' into 'master'
Move operations/ to new location ## What does this MR do? Move operations/ to new location. Part of https://gitlab.com/gitlab-org/gitlab-ce/issues/3349 ## Moving docs to a new location? See the guidelines: http://docs.gitlab.com/ce/development/doc_styleguide.html#changing-document-location - [ ] Make sure the old link is not removed and has its contents replaced with a link to the new location. - [ ] Make sure internal links pointing to the document in question are not broken. - [ ] Search and replace any links referring to old docs in GitLab Rails app, specifically under the `app/views/` directory. - [ ] If working on CE, submit an MR to EE with the changes as well. See merge request !6519
This commit is contained in:
commit
d078779174
|
@ -35,7 +35,7 @@
|
||||||
- [Libravatar](customization/libravatar.md) Use Libravatar instead of Gravatar for user avatars.
|
- [Libravatar](customization/libravatar.md) Use Libravatar instead of Gravatar for user avatars.
|
||||||
- [Log system](administration/logs.md) Log system.
|
- [Log system](administration/logs.md) Log system.
|
||||||
- [Environment Variables](administration/environment_variables.md) to configure GitLab.
|
- [Environment Variables](administration/environment_variables.md) to configure GitLab.
|
||||||
- [Operations](operations/README.md) Keeping GitLab up and running.
|
- [Operations](administration/operations.md) Keeping GitLab up and running.
|
||||||
- [Raketasks](raketasks/README.md) Backups, maintenance, automatic webhook setup and the importing of projects.
|
- [Raketasks](raketasks/README.md) Backups, maintenance, automatic webhook setup and the importing of projects.
|
||||||
- [Repository checks](administration/repository_checks.md) Periodic Git repository checks.
|
- [Repository checks](administration/repository_checks.md) Periodic Git repository checks.
|
||||||
- [Repository storages](administration/repository_storages.md) Manage the paths used to store repositories.
|
- [Repository storages](administration/repository_storages.md) Manage the paths used to store repositories.
|
||||||
|
|
|
@ -0,0 +1,6 @@
|
||||||
|
# GitLab operations
|
||||||
|
|
||||||
|
- [Sidekiq MemoryKiller](operations/sidekiq_memory_killer.md)
|
||||||
|
- [Cleaning up Redis sessions](operations/cleaning_up_redis_sessions.md)
|
||||||
|
- [Understanding Unicorn and unicorn-worker-killer](operations/unicorn.md)
|
||||||
|
- [Moving repositories to a new location](operations/moving_repositories.md)
|
|
@ -0,0 +1,52 @@
|
||||||
|
# Cleaning up stale Redis sessions
|
||||||
|
|
||||||
|
Since version 6.2, GitLab stores web user sessions as key-value pairs in Redis.
|
||||||
|
Prior to GitLab 7.3, user sessions did not automatically expire from Redis. If
|
||||||
|
you have been running a large GitLab server (thousands of users) since before
|
||||||
|
GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis
|
||||||
|
database after you upgrade to GitLab 7.3. You can also perform a cleanup while
|
||||||
|
still running GitLab 7.2 or older, but in that case new stale sessions will
|
||||||
|
start building up again after you clean up.
|
||||||
|
|
||||||
|
In GitLab versions prior to 7.3.0, the session keys in Redis are 16-byte
|
||||||
|
hexadecimal values such as '976aa289e2189b17d7ef525a6702ace9'. Starting with
|
||||||
|
GitLab 7.3.0, the keys are
|
||||||
|
prefixed with 'session:gitlab:', so they would look like
|
||||||
|
'session:gitlab:976aa289e2189b17d7ef525a6702ace9'. Below we describe how to
|
||||||
|
remove the keys in the old format.
|
||||||
|
|
||||||
|
First we define a shell function with the proper Redis connection details.
|
||||||
|
|
||||||
|
```
|
||||||
|
rcli() {
|
||||||
|
# This example works for Omnibus installations of GitLab 7.3 or newer. For an
|
||||||
|
# installation from source you will have to change the socket path and the
|
||||||
|
# path to redis-cli.
|
||||||
|
sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@"
|
||||||
|
}
|
||||||
|
|
||||||
|
# test the new shell function; the response should be PONG
|
||||||
|
rcli ping
|
||||||
|
```
|
||||||
|
|
||||||
|
Now we do a search to see if there are any session keys in the old format for
|
||||||
|
us to clean up.
|
||||||
|
|
||||||
|
```
|
||||||
|
# returns the number of old-format session keys in Redis
|
||||||
|
rcli keys '*' | grep '^[a-f0-9]\{32\}$' | wc -l
|
||||||
|
```
|
||||||
|
|
||||||
|
If the number is larger than zero, you can proceed to expire the keys from
|
||||||
|
Redis. If the number is zero there is nothing to clean up.
|
||||||
|
|
||||||
|
```
|
||||||
|
# Tell Redis to expire each matched key after 600 seconds.
|
||||||
|
rcli keys '*' | grep '^[a-f0-9]\{32\}$' | awk '{ print "expire", $0, 600 }' | rcli
|
||||||
|
# This will print '(integer) 1' for each key that gets expired.
|
||||||
|
```
|
||||||
|
|
||||||
|
Over the next 15 minutes (10 minutes expiry time plus 5 minutes Redis
|
||||||
|
background save interval) your Redis database will be compacted. If you are
|
||||||
|
still using GitLab 7.2, users who are not clicking around in GitLab during the
|
||||||
|
10 minute expiry window will be signed out of GitLab.
|
|
@ -0,0 +1,180 @@
|
||||||
|
# Moving repositories managed by GitLab
|
||||||
|
|
||||||
|
Sometimes you need to move all repositories managed by GitLab to
|
||||||
|
another filesystem or another server. In this document we will look
|
||||||
|
at some of the ways you can copy all your repositories from
|
||||||
|
`/var/opt/gitlab/git-data/repositories` to `/mnt/gitlab/repositories`.
|
||||||
|
|
||||||
|
We will look at three scenarios: the target directory is empty, the
|
||||||
|
target directory contains an outdated copy of the repositories, and
|
||||||
|
how to deal with thousands of repositories.
|
||||||
|
|
||||||
|
**Each of the approaches we list can/will overwrite data in the
|
||||||
|
target directory `/mnt/gitlab/repositories`. Do not mix up the
|
||||||
|
source and the target.**
|
||||||
|
|
||||||
|
## Target directory is empty: use a tar pipe
|
||||||
|
|
||||||
|
If the target directory `/mnt/gitlab/repositories` is empty the
|
||||||
|
simplest thing to do is to use a tar pipe. This method has low
|
||||||
|
overhead and tar is almost always already installed on your system.
|
||||||
|
However, it is not possible to resume an interrupted tar pipe: if
|
||||||
|
that happens then all data must be copied again.
|
||||||
|
|
||||||
|
```
|
||||||
|
# As the git user
|
||||||
|
tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\
|
||||||
|
tar -C /mnt/gitlab/repositories -xf -
|
||||||
|
```
|
||||||
|
|
||||||
|
If you want to see progress, replace `-xf` with `-xvf`.
|
||||||
|
|
||||||
|
### Tar pipe to another server
|
||||||
|
|
||||||
|
You can also use a tar pipe to copy data to another server. If your
|
||||||
|
'git' user has SSH access to the newserver as 'git@newserver', you
|
||||||
|
can pipe the data through SSH.
|
||||||
|
|
||||||
|
```
|
||||||
|
# As the git user
|
||||||
|
tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\
|
||||||
|
ssh git@newserver tar -C /mnt/gitlab/repositories -xf -
|
||||||
|
```
|
||||||
|
|
||||||
|
If you want to compress the data before it goes over the network
|
||||||
|
(which will cost you CPU cycles) you can replace `ssh` with `ssh -C`.
|
||||||
|
|
||||||
|
## The target directory contains an outdated copy of the repositories: use rsync
|
||||||
|
|
||||||
|
If the target directory already contains a partial / outdated copy
|
||||||
|
of the repositories it may be wasteful to copy all the data again
|
||||||
|
with tar. In this scenario it is better to use rsync. This utility
|
||||||
|
is either already installed on your system or easily installable
|
||||||
|
via apt, yum etc.
|
||||||
|
|
||||||
|
```
|
||||||
|
# As the 'git' user
|
||||||
|
rsync -a --delete /var/opt/gitlab/git-data/repositories/. \
|
||||||
|
/mnt/gitlab/repositories
|
||||||
|
```
|
||||||
|
|
||||||
|
The `/.` in the command above is very important, without it you can
|
||||||
|
easily get the wrong directory structure in the target directory.
|
||||||
|
If you want to see progress, replace `-a` with `-av`.
|
||||||
|
|
||||||
|
### Single rsync to another server
|
||||||
|
|
||||||
|
If the 'git' user on your source system has SSH access to the target
|
||||||
|
server you can send the repositories over the network with rsync.
|
||||||
|
|
||||||
|
```
|
||||||
|
# As the 'git' user
|
||||||
|
rsync -a --delete /var/opt/gitlab/git-data/repositories/. \
|
||||||
|
git@newserver:/mnt/gitlab/repositories
|
||||||
|
```
|
||||||
|
|
||||||
|
## Thousands of Git repositories: use one rsync per repository
|
||||||
|
|
||||||
|
Every time you start an rsync job it has to inspect all files in
|
||||||
|
the source directory, all files in the target directory, and then
|
||||||
|
decide what files to copy or not. If the source or target directory
|
||||||
|
has many contents this startup phase of rsync can become a burden
|
||||||
|
for your GitLab server. In cases like this you can make rsync's
|
||||||
|
life easier by dividing its work in smaller pieces, and sync one
|
||||||
|
repository at a time.
|
||||||
|
|
||||||
|
In addition to rsync we will use [GNU
|
||||||
|
Parallel](http://www.gnu.org/software/parallel/). This utility is
|
||||||
|
not included in GitLab so you need to install it yourself with apt
|
||||||
|
or yum. Also note that the GitLab scripts we used below were added
|
||||||
|
in GitLab 8.1.
|
||||||
|
|
||||||
|
** This process does not clean up repositories at the target location that no
|
||||||
|
longer exist at the source. ** If you start using your GitLab instance with
|
||||||
|
`/mnt/gitlab/repositories`, you need to run `gitlab-rake gitlab:cleanup:repos`
|
||||||
|
after switching to the new repository storage directory.
|
||||||
|
|
||||||
|
### Parallel rsync for all repositories known to GitLab
|
||||||
|
|
||||||
|
This will sync repositories with 10 rsync processes at a time. We keep
|
||||||
|
track of progress so that the transfer can be restarted if necessary.
|
||||||
|
|
||||||
|
First we create a new directory, owned by 'git', to hold transfer
|
||||||
|
logs. We assume the directory is empty before we start the transfer
|
||||||
|
procedure, and that we are the only ones writing files in it.
|
||||||
|
|
||||||
|
```
|
||||||
|
# Omnibus
|
||||||
|
sudo mkdir /var/opt/gitlab/transfer-logs
|
||||||
|
sudo chown git:git /var/opt/gitlab/transfer-logs
|
||||||
|
|
||||||
|
# Source
|
||||||
|
sudo -u git -H mkdir /home/git/transfer-logs
|
||||||
|
```
|
||||||
|
|
||||||
|
We seed the process with a list of the directories we want to copy.
|
||||||
|
|
||||||
|
```
|
||||||
|
# Omnibus
|
||||||
|
sudo -u git sh -c 'gitlab-rake gitlab:list_repos > /var/opt/gitlab/transfer-logs/all-repos-$(date +%s).txt'
|
||||||
|
|
||||||
|
# Source
|
||||||
|
cd /home/git/gitlab
|
||||||
|
sudo -u git -H sh -c 'bundle exec rake gitlab:list_repos > /home/git/transfer-logs/all-repos-$(date +%s).txt'
|
||||||
|
```
|
||||||
|
|
||||||
|
Now we can start the transfer. The command below is idempotent, and
|
||||||
|
the number of jobs done by GNU Parallel should converge to zero. If it
|
||||||
|
does not some repositories listed in all-repos-1234.txt may have been
|
||||||
|
deleted/renamed before they could be copied.
|
||||||
|
|
||||||
|
```
|
||||||
|
# Omnibus
|
||||||
|
sudo -u git sh -c '
|
||||||
|
cat /var/opt/gitlab/transfer-logs/* | sort | uniq -u |\
|
||||||
|
/usr/bin/env JOBS=10 \
|
||||||
|
/opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \
|
||||||
|
/var/opt/gitlab/transfer-logs/success-$(date +%s).log \
|
||||||
|
/var/opt/gitlab/git-data/repositories \
|
||||||
|
/mnt/gitlab/repositories
|
||||||
|
'
|
||||||
|
|
||||||
|
# Source
|
||||||
|
cd /home/git/gitlab
|
||||||
|
sudo -u git -H sh -c '
|
||||||
|
cat /home/git/transfer-logs/* | sort | uniq -u |\
|
||||||
|
/usr/bin/env JOBS=10 \
|
||||||
|
bin/parallel-rsync-repos \
|
||||||
|
/home/git/transfer-logs/success-$(date +%s).log \
|
||||||
|
/home/git/repositories \
|
||||||
|
/mnt/gitlab/repositories
|
||||||
|
`
|
||||||
|
```
|
||||||
|
|
||||||
|
### Parallel rsync only for repositories with recent activity
|
||||||
|
|
||||||
|
Suppose you have already done one sync that started after 2015-10-1 12:00 UTC.
|
||||||
|
Then you might only want to sync repositories that were changed via GitLab
|
||||||
|
_after_ that time. You can use the 'SINCE' variable to tell 'rake
|
||||||
|
gitlab:list_repos' to only print repositories with recent activity.
|
||||||
|
|
||||||
|
```
|
||||||
|
# Omnibus
|
||||||
|
sudo gitlab-rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\
|
||||||
|
sudo -u git \
|
||||||
|
/usr/bin/env JOBS=10 \
|
||||||
|
/opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \
|
||||||
|
success-$(date +%s).log \
|
||||||
|
/var/opt/gitlab/git-data/repositories \
|
||||||
|
/mnt/gitlab/repositories
|
||||||
|
|
||||||
|
# Source
|
||||||
|
cd /home/git/gitlab
|
||||||
|
sudo -u git -H bundle exec rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\
|
||||||
|
sudo -u git -H \
|
||||||
|
/usr/bin/env JOBS=10 \
|
||||||
|
bin/parallel-rsync-repos \
|
||||||
|
success-$(date +%s).log \
|
||||||
|
/home/git/repositories \
|
||||||
|
/mnt/gitlab/repositories
|
||||||
|
```
|
|
@ -0,0 +1,40 @@
|
||||||
|
# Sidekiq MemoryKiller
|
||||||
|
|
||||||
|
The GitLab Rails application code suffers from memory leaks. For web requests
|
||||||
|
this problem is made manageable using
|
||||||
|
[unicorn-worker-killer](https://github.com/kzk/unicorn-worker-killer) which
|
||||||
|
restarts Unicorn worker processes in between requests when needed. The Sidekiq
|
||||||
|
MemoryKiller applies the same approach to the Sidekiq processes used by GitLab
|
||||||
|
to process background jobs.
|
||||||
|
|
||||||
|
Unlike unicorn-worker-killer, which is enabled by default for all GitLab
|
||||||
|
installations since GitLab 6.4, the Sidekiq MemoryKiller is enabled by default
|
||||||
|
_only_ for Omnibus packages. The reason for this is that the MemoryKiller
|
||||||
|
relies on Runit to restart Sidekiq after a memory-induced shutdown and GitLab
|
||||||
|
installations from source do not all use Runit or an equivalent.
|
||||||
|
|
||||||
|
With the default settings, the MemoryKiller will cause a Sidekiq restart no
|
||||||
|
more often than once every 15 minutes, with the restart causing about one
|
||||||
|
minute of delay for incoming background jobs.
|
||||||
|
|
||||||
|
## Configuring the MemoryKiller
|
||||||
|
|
||||||
|
The MemoryKiller is controlled using environment variables.
|
||||||
|
|
||||||
|
- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is
|
||||||
|
greater than 0, then after each Sidekiq job, the MemoryKiller will check the
|
||||||
|
RSS of the Sidekiq process that executed the job. If the RSS of the Sidekiq
|
||||||
|
process (expressed in kilobytes) exceeds SIDEKIQ_MEMORY_KILLER_MAX_RSS, a
|
||||||
|
delayed shutdown is triggered. The default value for Omnibus packages is set
|
||||||
|
[in the omnibus-gitlab
|
||||||
|
repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb).
|
||||||
|
- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults 900 seconds (15 minutes). When
|
||||||
|
a shutdown is triggered, the Sidekiq process will keep working normally for
|
||||||
|
another 15 minutes.
|
||||||
|
- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. When the grace
|
||||||
|
time has expired, the MemoryKiller tells Sidekiq to stop accepting new jobs.
|
||||||
|
Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells
|
||||||
|
Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must
|
||||||
|
restart Sidekiq.
|
||||||
|
- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL`: defaults to `SIGKILL`. The name of
|
||||||
|
the final signal sent to the Sidekiq process when we want it to shut down.
|
|
@ -0,0 +1,86 @@
|
||||||
|
# Understanding Unicorn and unicorn-worker-killer
|
||||||
|
|
||||||
|
## Unicorn
|
||||||
|
|
||||||
|
GitLab uses [Unicorn](http://unicorn.bogomips.org/), a pre-forking Ruby web
|
||||||
|
server, to handle web requests (web browsers and Git HTTP clients). Unicorn is
|
||||||
|
a daemon written in Ruby and C that can load and run a Ruby on Rails
|
||||||
|
application; in our case the Rails application is GitLab Community Edition or
|
||||||
|
GitLab Enterprise Edition.
|
||||||
|
|
||||||
|
Unicorn has a multi-process architecture to make better use of available CPU
|
||||||
|
cores (processes can run on different cores) and to have stronger fault
|
||||||
|
tolerance (most failures stay isolated in only one process and cannot take down
|
||||||
|
GitLab entirely). On startup, the Unicorn 'master' process loads a clean Ruby
|
||||||
|
environment with the GitLab application code, and then spawns 'workers' which
|
||||||
|
inherit this clean initial environment. The 'master' never handles any
|
||||||
|
requests, that is left to the workers. The operating system network stack
|
||||||
|
queues incoming requests and distributes them among the workers.
|
||||||
|
|
||||||
|
In a perfect world, the master would spawn its pool of workers once, and then
|
||||||
|
the workers handle incoming web requests one after another until the end of
|
||||||
|
time. In reality, worker processes can crash or time out: if the master notices
|
||||||
|
that a worker takes too long to handle a request it will terminate the worker
|
||||||
|
process with SIGKILL ('kill -9'). No matter how the worker process ended, the
|
||||||
|
master process will replace it with a new 'clean' process again. Unicorn is
|
||||||
|
designed to be able to replace 'crashed' workers without dropping user
|
||||||
|
requests.
|
||||||
|
|
||||||
|
This is what a Unicorn worker timeout looks like in `unicorn_stderr.log`. The
|
||||||
|
master process has PID 56227 below.
|
||||||
|
|
||||||
|
```
|
||||||
|
[2015-06-05T10:58:08.660325 #56227] ERROR -- : worker=10 PID:53009 timeout (61s > 60s), killing
|
||||||
|
[2015-06-05T10:58:08.699360 #56227] ERROR -- : reaped #<Process::Status: pid 53009 SIGKILL (signal 9)> worker=10
|
||||||
|
[2015-06-05T10:58:08.708141 #62538] INFO -- : worker=10 spawned pid=62538
|
||||||
|
[2015-06-05T10:58:08.708824 #62538] INFO -- : worker=10 ready
|
||||||
|
```
|
||||||
|
|
||||||
|
### Tunables
|
||||||
|
|
||||||
|
The main tunables for Unicorn are the number of worker processes and the
|
||||||
|
request timeout after which the Unicorn master terminates a worker process.
|
||||||
|
See the [omnibus-gitlab Unicorn settings
|
||||||
|
documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md)
|
||||||
|
if you want to adjust these settings.
|
||||||
|
|
||||||
|
## unicorn-worker-killer
|
||||||
|
|
||||||
|
GitLab has memory leaks. These memory leaks manifest themselves in long-running
|
||||||
|
processes, such as Unicorn workers. (The Unicorn master process is not known to
|
||||||
|
leak memory, probably because it does not handle user requests.)
|
||||||
|
|
||||||
|
To make these memory leaks manageable, GitLab comes with the
|
||||||
|
[unicorn-worker-killer gem](https://github.com/kzk/unicorn-worker-killer). This
|
||||||
|
gem [monkey-patches](https://en.wikipedia.org/wiki/Monkey_patch) the Unicorn
|
||||||
|
workers to do a memory self-check after every 16 requests. If the memory of the
|
||||||
|
Unicorn worker exceeds a pre-set limit then the worker process exits. The
|
||||||
|
Unicorn master then automatically replaces the worker process.
|
||||||
|
|
||||||
|
This is a robust way to handle memory leaks: Unicorn is designed to handle
|
||||||
|
workers that 'crash' so no user requests will be dropped. The
|
||||||
|
unicorn-worker-killer gem is designed to only terminate a worker process _in
|
||||||
|
between requests_, so no user requests are affected.
|
||||||
|
|
||||||
|
This is what a Unicorn worker memory restart looks like in unicorn_stderr.log.
|
||||||
|
You see that worker 4 (PID 125918) is inspecting itself and decides to exit.
|
||||||
|
The threshold memory value was 254802235 bytes, about 250MB. With GitLab this
|
||||||
|
threshold is a random value between 200 and 250 MB. The master process (PID
|
||||||
|
117565) then reaps the worker process and spawns a new 'worker 4' with PID
|
||||||
|
127549.
|
||||||
|
|
||||||
|
```
|
||||||
|
[2015-06-05T12:07:41.828374 #125918] WARN -- : #<Unicorn::HttpServer:0x00000002734770>: worker (pid: 125918) exceeds memory limit (256413696 bytes > 254802235 bytes)
|
||||||
|
[2015-06-05T12:07:41.828472 #125918] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 125918) alive: 23 sec (trial 1)
|
||||||
|
[2015-06-05T12:07:42.025916 #117565] INFO -- : reaped #<Process::Status: pid 125918 exit 0> worker=4
|
||||||
|
[2015-06-05T12:07:42.034527 #127549] INFO -- : worker=4 spawned pid=127549
|
||||||
|
[2015-06-05T12:07:42.035217 #127549] INFO -- : worker=4 ready
|
||||||
|
```
|
||||||
|
|
||||||
|
One other thing that stands out in the log snippet above, taken from
|
||||||
|
GitLab.com, is that 'worker 4' was serving requests for only 23 seconds. This
|
||||||
|
is a normal value for our current GitLab.com setup and traffic.
|
||||||
|
|
||||||
|
The high frequency of Unicorn memory restarts on some GitLab sites can be a
|
||||||
|
source of confusion for administrators. Usually they are a [red
|
||||||
|
herring](https://en.wikipedia.org/wiki/Red_herring).
|
|
@ -1,5 +1 @@
|
||||||
# GitLab operations
|
This document was moved to [administration/operations](../administration/operations.md).
|
||||||
|
|
||||||
- [Sidekiq MemoryKiller](sidekiq_memory_killer.md)
|
|
||||||
- [Cleaning up Redis sessions](cleaning_up_redis_sessions.md)
|
|
||||||
- [Understanding Unicorn and unicorn-worker-killer](unicorn.md)
|
|
||||||
|
|
|
@ -1,52 +1 @@
|
||||||
# Cleaning up stale Redis sessions
|
This document was moved to [administration/operations/cleaning_up_redis_sessions](../administration/operations/cleaning_up_redis_sessions.md).
|
||||||
|
|
||||||
Since version 6.2, GitLab stores web user sessions as key-value pairs in Redis.
|
|
||||||
Prior to GitLab 7.3, user sessions did not automatically expire from Redis. If
|
|
||||||
you have been running a large GitLab server (thousands of users) since before
|
|
||||||
GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis
|
|
||||||
database after you upgrade to GitLab 7.3. You can also perform a cleanup while
|
|
||||||
still running GitLab 7.2 or older, but in that case new stale sessions will
|
|
||||||
start building up again after you clean up.
|
|
||||||
|
|
||||||
In GitLab versions prior to 7.3.0, the session keys in Redis are 16-byte
|
|
||||||
hexadecimal values such as '976aa289e2189b17d7ef525a6702ace9'. Starting with
|
|
||||||
GitLab 7.3.0, the keys are
|
|
||||||
prefixed with 'session:gitlab:', so they would look like
|
|
||||||
'session:gitlab:976aa289e2189b17d7ef525a6702ace9'. Below we describe how to
|
|
||||||
remove the keys in the old format.
|
|
||||||
|
|
||||||
First we define a shell function with the proper Redis connection details.
|
|
||||||
|
|
||||||
```
|
|
||||||
rcli() {
|
|
||||||
# This example works for Omnibus installations of GitLab 7.3 or newer. For an
|
|
||||||
# installation from source you will have to change the socket path and the
|
|
||||||
# path to redis-cli.
|
|
||||||
sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@"
|
|
||||||
}
|
|
||||||
|
|
||||||
# test the new shell function; the response should be PONG
|
|
||||||
rcli ping
|
|
||||||
```
|
|
||||||
|
|
||||||
Now we do a search to see if there are any session keys in the old format for
|
|
||||||
us to clean up.
|
|
||||||
|
|
||||||
```
|
|
||||||
# returns the number of old-format session keys in Redis
|
|
||||||
rcli keys '*' | grep '^[a-f0-9]\{32\}$' | wc -l
|
|
||||||
```
|
|
||||||
|
|
||||||
If the number is larger than zero, you can proceed to expire the keys from
|
|
||||||
Redis. If the number is zero there is nothing to clean up.
|
|
||||||
|
|
||||||
```
|
|
||||||
# Tell Redis to expire each matched key after 600 seconds.
|
|
||||||
rcli keys '*' | grep '^[a-f0-9]\{32\}$' | awk '{ print "expire", $0, 600 }' | rcli
|
|
||||||
# This will print '(integer) 1' for each key that gets expired.
|
|
||||||
```
|
|
||||||
|
|
||||||
Over the next 15 minutes (10 minutes expiry time plus 5 minutes Redis
|
|
||||||
background save interval) your Redis database will be compacted. If you are
|
|
||||||
still using GitLab 7.2, users who are not clicking around in GitLab during the
|
|
||||||
10 minute expiry window will be signed out of GitLab.
|
|
||||||
|
|
|
@ -1,180 +1 @@
|
||||||
# Moving repositories managed by GitLab
|
This document was moved to [administration/operations/moving_repositories](../administration/operations/moving_repositories.md).
|
||||||
|
|
||||||
Sometimes you need to move all repositories managed by GitLab to
|
|
||||||
another filesystem or another server. In this document we will look
|
|
||||||
at some of the ways you can copy all your repositories from
|
|
||||||
`/var/opt/gitlab/git-data/repositories` to `/mnt/gitlab/repositories`.
|
|
||||||
|
|
||||||
We will look at three scenarios: the target directory is empty, the
|
|
||||||
target directory contains an outdated copy of the repositories, and
|
|
||||||
how to deal with thousands of repositories.
|
|
||||||
|
|
||||||
**Each of the approaches we list can/will overwrite data in the
|
|
||||||
target directory `/mnt/gitlab/repositories`. Do not mix up the
|
|
||||||
source and the target.**
|
|
||||||
|
|
||||||
## Target directory is empty: use a tar pipe
|
|
||||||
|
|
||||||
If the target directory `/mnt/gitlab/repositories` is empty the
|
|
||||||
simplest thing to do is to use a tar pipe. This method has low
|
|
||||||
overhead and tar is almost always already installed on your system.
|
|
||||||
However, it is not possible to resume an interrupted tar pipe: if
|
|
||||||
that happens then all data must be copied again.
|
|
||||||
|
|
||||||
```
|
|
||||||
# As the git user
|
|
||||||
tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\
|
|
||||||
tar -C /mnt/gitlab/repositories -xf -
|
|
||||||
```
|
|
||||||
|
|
||||||
If you want to see progress, replace `-xf` with `-xvf`.
|
|
||||||
|
|
||||||
### Tar pipe to another server
|
|
||||||
|
|
||||||
You can also use a tar pipe to copy data to another server. If your
|
|
||||||
'git' user has SSH access to the newserver as 'git@newserver', you
|
|
||||||
can pipe the data through SSH.
|
|
||||||
|
|
||||||
```
|
|
||||||
# As the git user
|
|
||||||
tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\
|
|
||||||
ssh git@newserver tar -C /mnt/gitlab/repositories -xf -
|
|
||||||
```
|
|
||||||
|
|
||||||
If you want to compress the data before it goes over the network
|
|
||||||
(which will cost you CPU cycles) you can replace `ssh` with `ssh -C`.
|
|
||||||
|
|
||||||
## The target directory contains an outdated copy of the repositories: use rsync
|
|
||||||
|
|
||||||
If the target directory already contains a partial / outdated copy
|
|
||||||
of the repositories it may be wasteful to copy all the data again
|
|
||||||
with tar. In this scenario it is better to use rsync. This utility
|
|
||||||
is either already installed on your system or easily installable
|
|
||||||
via apt, yum etc.
|
|
||||||
|
|
||||||
```
|
|
||||||
# As the 'git' user
|
|
||||||
rsync -a --delete /var/opt/gitlab/git-data/repositories/. \
|
|
||||||
/mnt/gitlab/repositories
|
|
||||||
```
|
|
||||||
|
|
||||||
The `/.` in the command above is very important, without it you can
|
|
||||||
easily get the wrong directory structure in the target directory.
|
|
||||||
If you want to see progress, replace `-a` with `-av`.
|
|
||||||
|
|
||||||
### Single rsync to another server
|
|
||||||
|
|
||||||
If the 'git' user on your source system has SSH access to the target
|
|
||||||
server you can send the repositories over the network with rsync.
|
|
||||||
|
|
||||||
```
|
|
||||||
# As the 'git' user
|
|
||||||
rsync -a --delete /var/opt/gitlab/git-data/repositories/. \
|
|
||||||
git@newserver:/mnt/gitlab/repositories
|
|
||||||
```
|
|
||||||
|
|
||||||
## Thousands of Git repositories: use one rsync per repository
|
|
||||||
|
|
||||||
Every time you start an rsync job it has to inspect all files in
|
|
||||||
the source directory, all files in the target directory, and then
|
|
||||||
decide what files to copy or not. If the source or target directory
|
|
||||||
has many contents this startup phase of rsync can become a burden
|
|
||||||
for your GitLab server. In cases like this you can make rsync's
|
|
||||||
life easier by dividing its work in smaller pieces, and sync one
|
|
||||||
repository at a time.
|
|
||||||
|
|
||||||
In addition to rsync we will use [GNU
|
|
||||||
Parallel](http://www.gnu.org/software/parallel/). This utility is
|
|
||||||
not included in GitLab so you need to install it yourself with apt
|
|
||||||
or yum. Also note that the GitLab scripts we used below were added
|
|
||||||
in GitLab 8.1.
|
|
||||||
|
|
||||||
** This process does not clean up repositories at the target location that no
|
|
||||||
longer exist at the source. ** If you start using your GitLab instance with
|
|
||||||
`/mnt/gitlab/repositories`, you need to run `gitlab-rake gitlab:cleanup:repos`
|
|
||||||
after switching to the new repository storage directory.
|
|
||||||
|
|
||||||
### Parallel rsync for all repositories known to GitLab
|
|
||||||
|
|
||||||
This will sync repositories with 10 rsync processes at a time. We keep
|
|
||||||
track of progress so that the transfer can be restarted if necessary.
|
|
||||||
|
|
||||||
First we create a new directory, owned by 'git', to hold transfer
|
|
||||||
logs. We assume the directory is empty before we start the transfer
|
|
||||||
procedure, and that we are the only ones writing files in it.
|
|
||||||
|
|
||||||
```
|
|
||||||
# Omnibus
|
|
||||||
sudo mkdir /var/opt/gitlab/transfer-logs
|
|
||||||
sudo chown git:git /var/opt/gitlab/transfer-logs
|
|
||||||
|
|
||||||
# Source
|
|
||||||
sudo -u git -H mkdir /home/git/transfer-logs
|
|
||||||
```
|
|
||||||
|
|
||||||
We seed the process with a list of the directories we want to copy.
|
|
||||||
|
|
||||||
```
|
|
||||||
# Omnibus
|
|
||||||
sudo -u git sh -c 'gitlab-rake gitlab:list_repos > /var/opt/gitlab/transfer-logs/all-repos-$(date +%s).txt'
|
|
||||||
|
|
||||||
# Source
|
|
||||||
cd /home/git/gitlab
|
|
||||||
sudo -u git -H sh -c 'bundle exec rake gitlab:list_repos > /home/git/transfer-logs/all-repos-$(date +%s).txt'
|
|
||||||
```
|
|
||||||
|
|
||||||
Now we can start the transfer. The command below is idempotent, and
|
|
||||||
the number of jobs done by GNU Parallel should converge to zero. If it
|
|
||||||
does not some repositories listed in all-repos-1234.txt may have been
|
|
||||||
deleted/renamed before they could be copied.
|
|
||||||
|
|
||||||
```
|
|
||||||
# Omnibus
|
|
||||||
sudo -u git sh -c '
|
|
||||||
cat /var/opt/gitlab/transfer-logs/* | sort | uniq -u |\
|
|
||||||
/usr/bin/env JOBS=10 \
|
|
||||||
/opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \
|
|
||||||
/var/opt/gitlab/transfer-logs/success-$(date +%s).log \
|
|
||||||
/var/opt/gitlab/git-data/repositories \
|
|
||||||
/mnt/gitlab/repositories
|
|
||||||
'
|
|
||||||
|
|
||||||
# Source
|
|
||||||
cd /home/git/gitlab
|
|
||||||
sudo -u git -H sh -c '
|
|
||||||
cat /home/git/transfer-logs/* | sort | uniq -u |\
|
|
||||||
/usr/bin/env JOBS=10 \
|
|
||||||
bin/parallel-rsync-repos \
|
|
||||||
/home/git/transfer-logs/success-$(date +%s).log \
|
|
||||||
/home/git/repositories \
|
|
||||||
/mnt/gitlab/repositories
|
|
||||||
`
|
|
||||||
```
|
|
||||||
|
|
||||||
### Parallel rsync only for repositories with recent activity
|
|
||||||
|
|
||||||
Suppose you have already done one sync that started after 2015-10-1 12:00 UTC.
|
|
||||||
Then you might only want to sync repositories that were changed via GitLab
|
|
||||||
_after_ that time. You can use the 'SINCE' variable to tell 'rake
|
|
||||||
gitlab:list_repos' to only print repositories with recent activity.
|
|
||||||
|
|
||||||
```
|
|
||||||
# Omnibus
|
|
||||||
sudo gitlab-rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\
|
|
||||||
sudo -u git \
|
|
||||||
/usr/bin/env JOBS=10 \
|
|
||||||
/opt/gitlab/embedded/service/gitlab-rails/bin/parallel-rsync-repos \
|
|
||||||
success-$(date +%s).log \
|
|
||||||
/var/opt/gitlab/git-data/repositories \
|
|
||||||
/mnt/gitlab/repositories
|
|
||||||
|
|
||||||
# Source
|
|
||||||
cd /home/git/gitlab
|
|
||||||
sudo -u git -H bundle exec rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\
|
|
||||||
sudo -u git -H \
|
|
||||||
/usr/bin/env JOBS=10 \
|
|
||||||
bin/parallel-rsync-repos \
|
|
||||||
success-$(date +%s).log \
|
|
||||||
/home/git/repositories \
|
|
||||||
/mnt/gitlab/repositories
|
|
||||||
```
|
|
||||||
|
|
|
@ -1,40 +1 @@
|
||||||
# Sidekiq MemoryKiller
|
This document was moved to [administration/operations/sidekiq_memory_killer](../administration/operations/sidekiq_memory_killer.md).
|
||||||
|
|
||||||
The GitLab Rails application code suffers from memory leaks. For web requests
|
|
||||||
this problem is made manageable using
|
|
||||||
[unicorn-worker-killer](https://github.com/kzk/unicorn-worker-killer) which
|
|
||||||
restarts Unicorn worker processes in between requests when needed. The Sidekiq
|
|
||||||
MemoryKiller applies the same approach to the Sidekiq processes used by GitLab
|
|
||||||
to process background jobs.
|
|
||||||
|
|
||||||
Unlike unicorn-worker-killer, which is enabled by default for all GitLab
|
|
||||||
installations since GitLab 6.4, the Sidekiq MemoryKiller is enabled by default
|
|
||||||
_only_ for Omnibus packages. The reason for this is that the MemoryKiller
|
|
||||||
relies on Runit to restart Sidekiq after a memory-induced shutdown and GitLab
|
|
||||||
installations from source do not all use Runit or an equivalent.
|
|
||||||
|
|
||||||
With the default settings, the MemoryKiller will cause a Sidekiq restart no
|
|
||||||
more often than once every 15 minutes, with the restart causing about one
|
|
||||||
minute of delay for incoming background jobs.
|
|
||||||
|
|
||||||
## Configuring the MemoryKiller
|
|
||||||
|
|
||||||
The MemoryKiller is controlled using environment variables.
|
|
||||||
|
|
||||||
- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is
|
|
||||||
greater than 0, then after each Sidekiq job, the MemoryKiller will check the
|
|
||||||
RSS of the Sidekiq process that executed the job. If the RSS of the Sidekiq
|
|
||||||
process (expressed in kilobytes) exceeds SIDEKIQ_MEMORY_KILLER_MAX_RSS, a
|
|
||||||
delayed shutdown is triggered. The default value for Omnibus packages is set
|
|
||||||
[in the omnibus-gitlab
|
|
||||||
repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb).
|
|
||||||
- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults 900 seconds (15 minutes). When
|
|
||||||
a shutdown is triggered, the Sidekiq process will keep working normally for
|
|
||||||
another 15 minutes.
|
|
||||||
- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. When the grace
|
|
||||||
time has expired, the MemoryKiller tells Sidekiq to stop accepting new jobs.
|
|
||||||
Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells
|
|
||||||
Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must
|
|
||||||
restart Sidekiq.
|
|
||||||
- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL`: defaults to `SIGKILL`. The name of
|
|
||||||
the final signal sent to the Sidekiq process when we want it to shut down.
|
|
||||||
|
|
|
@ -1,86 +1 @@
|
||||||
# Understanding Unicorn and unicorn-worker-killer
|
This document was moved to [administration/operations/unicorn](../administration/operations/unicorn.md).
|
||||||
|
|
||||||
## Unicorn
|
|
||||||
|
|
||||||
GitLab uses [Unicorn](http://unicorn.bogomips.org/), a pre-forking Ruby web
|
|
||||||
server, to handle web requests (web browsers and Git HTTP clients). Unicorn is
|
|
||||||
a daemon written in Ruby and C that can load and run a Ruby on Rails
|
|
||||||
application; in our case the Rails application is GitLab Community Edition or
|
|
||||||
GitLab Enterprise Edition.
|
|
||||||
|
|
||||||
Unicorn has a multi-process architecture to make better use of available CPU
|
|
||||||
cores (processes can run on different cores) and to have stronger fault
|
|
||||||
tolerance (most failures stay isolated in only one process and cannot take down
|
|
||||||
GitLab entirely). On startup, the Unicorn 'master' process loads a clean Ruby
|
|
||||||
environment with the GitLab application code, and then spawns 'workers' which
|
|
||||||
inherit this clean initial environment. The 'master' never handles any
|
|
||||||
requests, that is left to the workers. The operating system network stack
|
|
||||||
queues incoming requests and distributes them among the workers.
|
|
||||||
|
|
||||||
In a perfect world, the master would spawn its pool of workers once, and then
|
|
||||||
the workers handle incoming web requests one after another until the end of
|
|
||||||
time. In reality, worker processes can crash or time out: if the master notices
|
|
||||||
that a worker takes too long to handle a request it will terminate the worker
|
|
||||||
process with SIGKILL ('kill -9'). No matter how the worker process ended, the
|
|
||||||
master process will replace it with a new 'clean' process again. Unicorn is
|
|
||||||
designed to be able to replace 'crashed' workers without dropping user
|
|
||||||
requests.
|
|
||||||
|
|
||||||
This is what a Unicorn worker timeout looks like in `unicorn_stderr.log`. The
|
|
||||||
master process has PID 56227 below.
|
|
||||||
|
|
||||||
```
|
|
||||||
[2015-06-05T10:58:08.660325 #56227] ERROR -- : worker=10 PID:53009 timeout (61s > 60s), killing
|
|
||||||
[2015-06-05T10:58:08.699360 #56227] ERROR -- : reaped #<Process::Status: pid 53009 SIGKILL (signal 9)> worker=10
|
|
||||||
[2015-06-05T10:58:08.708141 #62538] INFO -- : worker=10 spawned pid=62538
|
|
||||||
[2015-06-05T10:58:08.708824 #62538] INFO -- : worker=10 ready
|
|
||||||
```
|
|
||||||
|
|
||||||
### Tunables
|
|
||||||
|
|
||||||
The main tunables for Unicorn are the number of worker processes and the
|
|
||||||
request timeout after which the Unicorn master terminates a worker process.
|
|
||||||
See the [omnibus-gitlab Unicorn settings
|
|
||||||
documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md)
|
|
||||||
if you want to adjust these settings.
|
|
||||||
|
|
||||||
## unicorn-worker-killer
|
|
||||||
|
|
||||||
GitLab has memory leaks. These memory leaks manifest themselves in long-running
|
|
||||||
processes, such as Unicorn workers. (The Unicorn master process is not known to
|
|
||||||
leak memory, probably because it does not handle user requests.)
|
|
||||||
|
|
||||||
To make these memory leaks manageable, GitLab comes with the
|
|
||||||
[unicorn-worker-killer gem](https://github.com/kzk/unicorn-worker-killer). This
|
|
||||||
gem [monkey-patches](https://en.wikipedia.org/wiki/Monkey_patch) the Unicorn
|
|
||||||
workers to do a memory self-check after every 16 requests. If the memory of the
|
|
||||||
Unicorn worker exceeds a pre-set limit then the worker process exits. The
|
|
||||||
Unicorn master then automatically replaces the worker process.
|
|
||||||
|
|
||||||
This is a robust way to handle memory leaks: Unicorn is designed to handle
|
|
||||||
workers that 'crash' so no user requests will be dropped. The
|
|
||||||
unicorn-worker-killer gem is designed to only terminate a worker process _in
|
|
||||||
between requests_, so no user requests are affected.
|
|
||||||
|
|
||||||
This is what a Unicorn worker memory restart looks like in unicorn_stderr.log.
|
|
||||||
You see that worker 4 (PID 125918) is inspecting itself and decides to exit.
|
|
||||||
The threshold memory value was 254802235 bytes, about 250MB. With GitLab this
|
|
||||||
threshold is a random value between 200 and 250 MB. The master process (PID
|
|
||||||
117565) then reaps the worker process and spawns a new 'worker 4' with PID
|
|
||||||
127549.
|
|
||||||
|
|
||||||
```
|
|
||||||
[2015-06-05T12:07:41.828374 #125918] WARN -- : #<Unicorn::HttpServer:0x00000002734770>: worker (pid: 125918) exceeds memory limit (256413696 bytes > 254802235 bytes)
|
|
||||||
[2015-06-05T12:07:41.828472 #125918] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 125918) alive: 23 sec (trial 1)
|
|
||||||
[2015-06-05T12:07:42.025916 #117565] INFO -- : reaped #<Process::Status: pid 125918 exit 0> worker=4
|
|
||||||
[2015-06-05T12:07:42.034527 #127549] INFO -- : worker=4 spawned pid=127549
|
|
||||||
[2015-06-05T12:07:42.035217 #127549] INFO -- : worker=4 ready
|
|
||||||
```
|
|
||||||
|
|
||||||
One other thing that stands out in the log snippet above, taken from
|
|
||||||
GitLab.com, is that 'worker 4' was serving requests for only 23 seconds. This
|
|
||||||
is a normal value for our current GitLab.com setup and traffic.
|
|
||||||
|
|
||||||
The high frequency of Unicorn memory restarts on some GitLab sites can be a
|
|
||||||
source of confusion for administrators. Usually they are a [red
|
|
||||||
herring](https://en.wikipedia.org/wiki/Red_herring).
|
|
||||||
|
|
Loading…
Reference in New Issue