kotovalexarian-likes-gitlab/gitlab-org--gitlab-foss
1
0
Fork 0
Code Releases Activity
gitlab-org--gitlab-foss/doc/administration/high_availability/gitlab.md

207 lines
9.3 KiB
Markdown
Raw Normal View History

SSoT for administration/high_availability docs - Make sure we have type defined in the frontmatter - Add troubleshooting sections where absent - Meaningful intros where absent
2019-07-15 23:32:29 -04:00
---
type: reference
---
# Configuring GitLab for Scaling and High Availability
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
> **Note:** There is some additional configuration near the bottom for
Update server terminolgy to differentiate HA setups from Geo
2017-07-27 16:33:09 -04:00
additional GitLab application servers. It's important to read and understand
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
these additional steps before proceeding with GitLab installation.
1. If necessary, install the NFS client utility packages using the following
commands:
Fix whitespace in many administration docs Many code blocks are 4spaced, and they render in GitLab without coloring as a result, even though they are fenced with a language label. If in a list, other items will render as being in a code block too, even if not meant to. This fixes all these issues for many admin docs in /high_availability and /auth (part 1)
2019-07-11 22:06:46 -04:00
```
# Ubuntu/Debian
apt-get install nfs-common
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
Fix whitespace in many administration docs Many code blocks are 4spaced, and they render in GitLab without coloring as a result, even though they are fenced with a language label. If in a list, other items will render as being in a code block too, even if not meant to. This fixes all these issues for many admin docs in /high_availability and /auth (part 1)
2019-07-11 22:06:46 -04:00
# CentOS/Red Hat
yum install nfs-utils nfs-utils-lib
```
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
1. Specify the necessary NFS shares. Mounts are specified in
`/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose
to configure your NFS server. See [NFS documentation](nfs.md) for the various
options. Here is an example snippet to add to `/etc/fstab`:
Fix whitespace in many administration docs Many code blocks are 4spaced, and they render in GitLab without coloring as a result, even though they are fenced with a language label. If in a list, other items will render as being in a code block too, even if not meant to. This fixes all these issues for many admin docs in /high_availability and /auth (part 1)
2019-07-11 22:06:46 -04:00
```
10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
```
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
1. Create the shared directories. These may be different depending on your NFS
mount locations.
Fix whitespace in many administration docs Many code blocks are 4spaced, and they render in GitLab without coloring as a result, even though they are fenced with a language label. If in a list, other items will render as being in a code block too, even if not meant to. This fixes all these issues for many admin docs in /high_availability and /auth (part 1)
2019-07-11 22:06:46 -04:00
```
mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data
```
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
1. Download/install GitLab Omnibus using **steps 1 and 2** from
[GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
steps on the download page.
1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
Be sure to change the `external_url` to match your eventual GitLab front-end
URL. Depending your the NFS configuration, you may need to change some GitLab
data locations. See [NFS documentation](nfs.md) for `/etc/gitlab/gitlab.rb`
configuration values for various scenarios. The example below assumes you've
ensure UIDs and GIDs match for HA setup
2018-05-29 08:28:52 -04:00
added NFS mounts in the default data locations. Additionally the UID and GIDs
given are just examples and you should configure with your preferred values.
Tidy up the documentation of Gitlab HA/Gitlab Application
2017-12-19 09:53:27 -05:00
Fix whitespace in many administration docs Many code blocks are 4spaced, and they render in GitLab without coloring as a result, even though they are fenced with a language label. If in a list, other items will render as being in a code block too, even if not meant to. This fixes all these issues for many admin docs in /high_availability and /auth (part 1)
2019-07-11 22:06:46 -04:00
```ruby
external_url 'https://gitlab.example.com'
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
Fix whitespace in many administration docs Many code blocks are 4spaced, and they render in GitLab without coloring as a result, even though they are fenced with a language label. If in a list, other items will render as being in a code block too, even if not meant to. This fixes all these issues for many admin docs in /high_availability and /auth (part 1)
2019-07-11 22:06:46 -04:00
# Prevent GitLab from starting if NFS data mounts are not available
high_availability['mountpoint'] = '/var/opt/gitlab/git-data'
Tidy up the documentation of Gitlab HA/Gitlab Application
2017-12-19 09:53:27 -05:00
Fix whitespace in many administration docs Many code blocks are 4spaced, and they render in GitLab without coloring as a result, even though they are fenced with a language label. If in a list, other items will render as being in a code block too, even if not meant to. This fixes all these issues for many admin docs in /high_availability and /auth (part 1)
2019-07-11 22:06:46 -04:00
# Disable components that will not be on the GitLab application server
roles ['application_role']
nginx['enable'] = true
Tidy up the documentation of Gitlab HA/Gitlab Application
2017-12-19 09:53:27 -05:00
Fix whitespace in many administration docs Many code blocks are 4spaced, and they render in GitLab without coloring as a result, even though they are fenced with a language label. If in a list, other items will render as being in a code block too, even if not meant to. This fixes all these issues for many admin docs in /high_availability and /auth (part 1)
2019-07-11 22:06:46 -04:00
# PostgreSQL connection details
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server
gitlab_rails['db_password'] = 'DB password'
Tidy up the documentation of Gitlab HA/Gitlab Application
2017-12-19 09:53:27 -05:00
Fix whitespace in many administration docs Many code blocks are 4spaced, and they render in GitLab without coloring as a result, even though they are fenced with a language label. If in a list, other items will render as being in a code block too, even if not meant to. This fixes all these issues for many admin docs in /high_availability and /auth (part 1)
2019-07-11 22:06:46 -04:00
# Redis connection details
gitlab_rails['redis_port'] = '6379'
gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server
gitlab_rails['redis_password'] = 'Redis Password'
Https many links in docs
2019-06-18 08:23:07 -04:00
Fix whitespace in many administration docs Many code blocks are 4spaced, and they render in GitLab without coloring as a result, even though they are fenced with a language label. If in a list, other items will render as being in a code block too, even if not meant to. This fixes all these issues for many admin docs in /high_availability and /auth (part 1)
2019-07-11 22:06:46 -04:00
# Ensure UIDs and GIDs match between servers for permissions via NFS
user['uid'] = 9000
user['gid'] = 9000
web_server['uid'] = 9001
web_server['gid'] = 9001
registry['uid'] = 9002
registry['gid'] = 9002
```
Tidy up the documentation of Gitlab HA/Gitlab Application
2017-12-19 09:53:27 -05:00
Adding documentation for Prometheus Service Discovery Updating documentation to include Prometheus Service Discovery. Added monitoring node documentaiton as well as documentation for the individual service nodes.
2019-06-21 11:55:57 -04:00
1. [Enable monitoring](#enable-monitoring)
Fix whitespace in many administration docs Many code blocks are 4spaced, and they render in GitLab without coloring as a result, even though they are fenced with a language label. If in a list, other items will render as being in a code block too, even if not meant to. This fixes all these issues for many admin docs in /high_availability and /auth (part 1)
2019-07-11 22:06:46 -04:00
> **Note:** To maintain uniformity of links across HA clusters, the `external_url`
on the first application server as well as the additional application
servers should point to the external url that users will use to access GitLab.
In a typical HA setup, this will be the url of the load balancer which will
route traffic to all GitLab application servers in the HA cluster.
>
> **Note:** When you specify `https` in the `external_url`, as in the example
above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If
certificates are not present, Nginx will fail to start. See
[Nginx documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https)
for more information.
>
> **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure
of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure.
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
Update server terminolgy to differentiate HA setups from Geo
2017-07-27 16:33:09 -04:00
## First GitLab application server
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
cleanup application server setup
2018-05-28 08:12:05 -04:00
As a final step, run the setup rake task **only on** the first GitLab application server.
Do not run this on additional application servers.
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
1. Initialize the database by running `sudo gitlab-rake gitlab:setup`.
cleanup application server setup
2018-05-28 08:12:05 -04:00
1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
> **WARNING:** Only run this setup task on **NEW** GitLab instances because it
will wipe any existing data.
Update server terminolgy to differentiate HA setups from Geo
2017-07-27 16:33:09 -04:00
## Extra configuration for additional GitLab application servers
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
Update server terminolgy to differentiate HA setups from Geo
2017-07-27 16:33:09 -04:00
Additional GitLab servers (servers configured **after** the first GitLab server)
need some extra configuration.
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
1. Configure shared secrets. These values can be obtained from the primary
Docs: Realign several CE docs that diverged from EE unnecessarily
2019-02-12 07:34:21 -05:00
GitLab server in `/etc/gitlab/gitlab-secrets.json`. Copy this file to the
secondary servers **prior to** running the first `reconfigure` in the steps
above.
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
Fix whitespace in many administration docs Many code blocks are 4spaced, and they render in GitLab without coloring as a result, even though they are fenced with a language label. If in a list, other items will render as being in a code block too, even if not meant to. This fixes all these issues for many admin docs in /high_availability and /auth (part 1)
2019-07-11 22:06:46 -04:00
```ruby
gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860'
gitlab_rails['otp_key_base'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa'
gitlab_rails['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d'
gitlab_rails['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964'
```
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
Replace deprecated skip-auto-migrations occurrence
2018-11-23 09:44:27 -05:00
1. Run `touch /etc/gitlab/skip-auto-reconfigure` to prevent database migrations
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
from running on upgrade. Only the primary GitLab application server should
handle migrations.
Makes copying host keys a recommended step
2019-07-09 02:23:17 -04:00
1. **Recommended** Configure host keys. Copy the contents (primary and public keys) of `/etc/ssh/` on
Update gitlab.md
2018-07-13 12:39:55 -04:00
the primary application server to `/etc/ssh` on all secondary servers. This
prevents false man-in-the-middle-attack alerts when accessing servers in your
High Availability cluster behind a load balancer.
cleanup application server setup
2018-05-28 08:12:05 -04:00
1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
Adding documentation for Prometheus Service Discovery Updating documentation to include Prometheus Service Discovery. Added monitoring node documentaiton as well as documentation for the individual service nodes.
2019-06-21 11:55:57 -04:00
## Enable Monitoring
> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.
If you enable Monitoring, it must be enabled on **all** GitLab servers.
Add note to collect CONSUL_SERVER_NODES in HA docs There are several instances in our HA docs where an example of `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` is given, but this value isn't explicitly stated. It should be the IP addresses or DNS records of the Consul server nodes.
2019-06-21 16:41:51 -04:00
1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z`
Adding documentation for Prometheus Service Discovery Updating documentation to include Prometheus Service Discovery. Added monitoring node documentaiton as well as documentation for the individual service nodes.
2019-06-21 11:55:57 -04:00
1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:
```ruby
# Enable service discovery for Prometheus
consul['enable'] = true
consul['monitoring_service_discovery'] = true
# Replace placeholders
# Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
# with the addresses of the Consul server nodes
consul['configuration'] = {
retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
}
# Set the network addresses that the exporters will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229'
sidekiq['listen_address'] = "0.0.0.0"
unicorn['listen'] = '0.0.0.0'
Update HA monitoring documentation * Update wording and include localhost for monitoring whitelist configurations. * Add monitoring config documentation for pgbouncer_exporter.
2019-06-26 03:57:20 -04:00
# Add the monitoring node's IP address to the monitoring whitelist and allow it to
# scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with
# the address and/or subnets gathered from the monitoring node(s).
gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com', '127.0.0.0/8']
nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com', '127.0.0.0/8']
Adding documentation for Prometheus Service Discovery Updating documentation to include Prometheus Service Discovery. Added monitoring node documentaiton as well as documentation for the individual service nodes.
2019-06-21 11:55:57 -04:00
```
1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
> **Warning:** After changing `unicorn['listen']` in `gitlab.rb`, and running `sudo gitlab-ctl reconfigure`,
it can take an extended period of time for unicorn to complete reloading after receiving a `HUP`.
For more information, see the [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4401).
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
## Troubleshooting
- `mount: wrong fs type, bad option, bad superblock on`
You have not installed the necessary NFS client utilities. See step 1 above.
- `mount: mount point /var/opt/gitlab/... does not exist`
This particular directory does not exist on the NFS server. Ensure
the share is exported and exists on the NFS server and try to remount.
---
Add documentation on upgrading GitLab HA nodes As mentioned in https://gitlab.com/gitlab-org/gitlab-ce/issues/58121#note_145299901, it wasn't obvious that upgrading an GitLab HA cluster required special care. Link to the Omnibus documentation for upgrade instructions. Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/58121
2019-02-26 14:30:43 -05:00
## Upgrading GitLab HA
GitLab HA installations can be upgraded with no downtime, but the
upgrade process must be carefully coordinated to avoid failures. See the
[Omnibus GitLab multi-node upgrade
document](https://docs.gitlab.com/omnibus/update/#multi-node--ha-deployment)
for more details.
New HA docs [ci skip]
2016-04-22 14:18:06 -04:00
Read more on high-availability configuration:
1. [Configure the database](database.md)
1. [Configure Redis](redis.md)
1. [Configure NFS](nfs.md)
1. [Configure the load balancers](load_balancer.md)
Copy permalink