Add parallel-rsync-repos script and start docs

bin/parallel-rsync-repos

@@ -0,0 +1,26 @@
+#!/bin/sh
+# this script should run as the 'git' user, not root, because of mkdir
+#
+# Example invocation:
+# find /var/opt/gitlab/git-data/repositories -maxdepth 2 | \
+#   parallel-rsync-repos /var/opt/gitlab/git-data/repositories /mnt/gitlab/repositories
+
+SRC=$1
+DEST=$2
+
+if [ -z "$JOBS" ] ; then
+  JOBS=10
+fi
+
+if [ -z "$SRC" ] || [ -z "$DEST" ] ; then
+  echo "Usage: $0 SRC DEST"
+  exit 1
+fi
+
+if ! cd $SRC ; then
+  echo "cd $SRC failed"
+  exit 1
+fi
+
+sed "s|$SRC|./|" |\
+  parallel -j$JOBS --progress "mkdir -p $DEST/{} && rsync --delete -a {}/. $DEST/{}/"

doc/operations/rsyncing_repositories.md

@@ -0,0 +1,87 @@
+# Moving repositories managed by GitLab
+
+Sometimes you need to move all repositories managed by GitLab to
+another filesystem or another server. In this document we will look
+at some of the ways you can copy all your repositories from
+`/var/opt/gitlab/git-data/repositories` to `/mnt/gitlab/repositories`.
+
+We will look at three scenarios: the target directory is empty, the
+target directory contains an outdated copy of the repositories, and
+how to deal with thousands of repositories.
+
+**Each of the approaches we list can/will overwrite data in the
+target directory `/mnt/gitlab/repositories`. Do not mix up the
+source and the target.**
+
+## Target directory is empty: use a tar pipe
+
+If the target directory `/mnt/gitlab/repositories` is empty the
+simplest thing to do is to use a tar pipe.
+
+```
+# As the git user
+tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\
+  tar -C /mnt/gitlab/repositories -xf -
+```
+
+If you want to see progress, replace `-xf` with `-xvf`.
+
+### Tar pipe to another server
+
+You can also use a tar pipe to copy data to another server. If your
+'git' user has SSH access to the newserver as 'git@newserver', you
+can pipe the data through SSH.
+
+```
+# As the git user
+tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\
+  ssh git@newserver tar -C /mnt/gitlab/repositories -xf -
+```
+
+If you want to compress the data before it goes over the network
+(which will cost you CPU cycles) you can replace `ssh` with `ssh
+-C`.
+
+## The target directory contains an outdated copy of the repositories: use rsync
+
+In this scenario it is better to use rsync. This utility is either
+already installed on your system or easily installable via apt, yum
+etc.
+
+```
+# As the 'git' user
+rsync -a --delete /var/opt/gitlab/git-data/repositories/. \
+  /mnt/gitlab/repositories
+```
+
+The `/.` in the command above is very important, without it you can
+easily get the wrong directory structure in the target directory.
+If you want to see progress, replace `-a` with `-av`.
+
+### Single rsync to another server
+
+If the 'git' user on your source system has SSH access to the target
+server you can send the repositories over the network with rsync.
+
+```
+# As the 'git' user
+rsync -a --delete /var/opt/gitlab/git-data/repositories/. \
+  git@newserver:/mnt/gitlab/repositories
+```
+
+## Thousands of Git repositories: use one rsync per repository
+
+Every time you start an rsync job it has to inspect all files in
+the source directory, all files in the target directory, and then
+decide what files to copy or not. If the source or target directory
+has many contents this startup phase of rsync can become a burden
+for your GitLab server. In cases like this you can make rsync's
+life easier by dividing its work in smaller pieces, and sync one
+repository at a time.
+
+In addition to rsync we will use [GNU
+Parallel](http://www.gnu.org/software/parallel/). This utility is
+not included in GitLab so you need to install it yourself with apt
+or yum.  Also note that the GitLab scripts we used below were added
+in GitLab 8.???.
+