diff --git a/.gitlab/ci/frontend.gitlab-ci.yml b/.gitlab/ci/frontend.gitlab-ci.yml index da74f4de6c2..1538352a881 100644 --- a/.gitlab/ci/frontend.gitlab-ci.yml +++ b/.gitlab/ci/frontend.gitlab-ci.yml @@ -239,7 +239,9 @@ jest: parallel: 2 jest-integration: - extends: .frontend-job-base + extends: + - .frontend-job-base + - .frontend:rules:default-frontend-jobs script: - date - yarn jest:integration --ci diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md index 9322c4cc417..214f15b7565 100644 --- a/doc/administration/geo/replication/high_availability.md +++ b/doc/administration/geo/replication/high_availability.md @@ -1,459 +1,5 @@ -# Geo for multiple servers **(PREMIUM ONLY)** +--- +redirect_to: 'multiple_servers.md' +--- -This document describes a minimal reference architecture for running Geo -in a multi-server configuration. If your multi-server setup differs from the one -described, it is possible to adapt these instructions to your needs. - -## Architecture overview - -![Geo multi-server diagram](../../high_availability/img/geo-ha-diagram.png) - -_[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_ - -The topology above assumes that the **primary** and **secondary** Geo clusters -are located in two separate locations, on their own virtual network -with private IP addresses. The network is configured such that all machines within -one geographic location can communicate with each other using their private IP addresses. -The IP addresses given are examples and may be different depending on the -network topology of your deployment. - -The only external way to access the two Geo deployments is by HTTPS at -`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. - -NOTE: **Note:** -The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS. - -## Redis and PostgreSQL for multiple servers - -Geo supports: - -- Redis and PostgreSQL on the **primary** node configured for multiple servers. -- Redis on **secondary** nodes configured for multiple servers. - -NOTE: **Note:** -Support for PostgreSQL on **secondary** nodes in multi-server configuration -[is planned](https://gitlab.com/groups/gitlab-org/-/epics/2536). - -Because of the additional complexity involved in setting up this configuration -for PostgreSQL and Redis, it is not covered by this Geo multi-server documentation. - -For more information about setting up a multi-server PostgreSQL cluster and Redis cluster using the omnibus package see the multi-server documentation for -[PostgreSQL](../../high_availability/database.md) and -[Redis](../../high_availability/redis.md), respectively. - -NOTE: **Note:** -It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. - -## Prerequisites: Two working GitLab multi-server clusters - -One cluster will serve as the **primary** node. Use the -[GitLab multi-server documentation](../../reference_architectures/index.md) to set this up. If -you already have a working GitLab instance that is in-use, it can be used as a -**primary**. - -The second cluster will serve as the **secondary** node. Again, use the -[GitLab multi-server documentation](../../reference_architectures/index.md) to set this up. -It's a good idea to log in and test it, however, note that its data will be -wiped out as part of the process of replicating from the **primary**. - -## Configure the GitLab cluster to be the **primary** node - -The following steps enable a GitLab cluster to serve as the **primary** node. - -### Step 1: Configure the **primary** frontend servers - -1. Edit `/etc/gitlab/gitlab.rb` and add the following: - - ```ruby - ## - ## Enable the Geo primary role - ## - roles ['geo_primary_role'] - - ## - ## The unique identifier for the Geo node. - ## - gitlab_rails['geo_node_name'] = '' - - ## - ## Disable automatic migrations - ## - gitlab_rails['auto_migrate'] = false - ``` - -After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. - -NOTE: **Note:** PostgreSQL and Redis should have already been disabled on the -application servers, and connections from the application servers to those -services on the backend servers configured, during normal GitLab multi-server set up. See -multi-server configuration documentation for -[PostgreSQL](../../high_availability/database.md#configuring-the-application-nodes) -and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitlab-application). - -### Step 2: Configure the **primary** database - -1. Edit `/etc/gitlab/gitlab.rb` and add the following: - - ```ruby - ## - ## Configure the Geo primary role and the PostgreSQL role - ## - roles ['geo_primary_role', 'postgres_role'] - ``` - -## Configure a **secondary** node - -A **secondary** cluster is similar to any other GitLab multi-server cluster, with two -major differences: - -- The main PostgreSQL database is a read-only replica of the **primary** node's - PostgreSQL database. -- There is also a single PostgreSQL database for the **secondary** cluster, - called the "tracking database", which tracks the synchronization state of - various resources. - -Therefore, we will set up the multi-server components one-by-one, and include deviations -from the normal multi-server setup. However, we highly recommend first configuring a -brand-new cluster as if it were not part of a Geo setup so that it can be -tested and verified as a working cluster. And only then should it be modified -for use as a Geo **secondary**. This helps to separate problems that are related -and are not related to Geo setup. - -### Step 1: Configure the Redis and Gitaly services on the **secondary** node - -Configure the following services, again using the non-Geo multi-server -documentation: - -- [Configuring Redis for GitLab](../../high_availability/redis.md) for multiple servers. -- [Gitaly](../../high_availability/gitaly.md), which will store data that is - synchronized from the **primary** node. - -NOTE: **Note:** -[NFS](../../high_availability/nfs.md) can be used in place of Gitaly but is not -recommended. - -### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node - -NOTE: **Note:** The following documentation assumes the database will be run on -a single node only. Multi-server PostgreSQL on **secondary** nodes is -[not currently supported](https://gitlab.com/groups/gitlab-org/-/epics/2536). - -Configure the [**secondary** database](database.md) as a read-only replica of -the **primary** database. Use the following as a guide. - -1. Generate an MD5 hash of the desired password for the database user that the - GitLab application will use to access the read-replica database: - - Note that the username (`gitlab` by default) is incorporated into the hash. - - ```shell - gitlab-ctl pg-password-md5 gitlab - # Enter password: - # Confirm password: - # fca0b89a972d69f00eb3ec98a5838484 - ``` - - Use this hash to fill in `` in the next step. - -1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the - following: - - ```ruby - ## - ## Configure the Geo secondary role and the PostgreSQL role - ## - roles ['geo_secondary_role', 'postgres_role'] - - ## - ## Secondary address - ## - replace '' with the public or VPC address of your Geo secondary node - ## - replace '' with the public or VPC address of your Geo tracking database node - ## - postgresql['listen_address'] = '' - postgresql['md5_auth_cidr_addresses'] = ['/32', '/32'] - - ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node - ## - postgresql['sql_user_password'] = '' - gitlab_rails['db_password'] = '' - - ## - ## When running the Geo tracking database on a separate machine, disable it - ## here and allow connections from the tracking database host. And ensure - ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. - ## - geo_postgresql['enable'] = false - - ## - ## Disable `geo_logcursor` service so Rails doesn't get configured here - ## - geo_logcursor['enable'] = false - ``` - -After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. - -If using an external PostgreSQL instance, refer also to -[Geo with external PostgreSQL instances](external_database.md). - -### Step 3: Configure the tracking database on the **secondary** node - -NOTE: **Note:** This documentation assumes the tracking database will be run on -only a single machine, rather than as a PostgreSQL cluster. - -Configure the tracking database. - -1. Generate an MD5 hash of the desired password for the database user that the - GitLab application will use to access the tracking database: - - Note that the username (`gitlab_geo` by default) is incorporated into the - hash. - - ```shell - gitlab-ctl pg-password-md5 gitlab_geo - # Enter password: - # Confirm password: - # fca0b89a972d69f00eb3ec98a5838484 - ``` - - Use this hash to fill in `` in the next - step. - -1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the - following: - - ```ruby - ## - ## Enable the Geo secondary tracking database - ## - geo_postgresql['enable'] = true - geo_postgresql['listen_address'] = '' - geo_postgresql['sql_user_password'] = '' - - ## - ## Configure FDW connection to the replica database - ## - geo_secondary['db_fdw'] = true - geo_postgresql['fdw_external_password'] = '' - geo_postgresql['md5_auth_cidr_addresses'] = ['/32'] - gitlab_rails['db_host'] = '' - - # Prevent reconfigure from attempting to run migrations on the replica DB - gitlab_rails['auto_migrate'] = false - - ## - ## Disable all other services that aren't needed, since we don't have a role - ## that does this. - ## - alertmanager['enable'] = false - consul['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - postgresql['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - repmgr['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - ``` - -After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. - -If using an external PostgreSQL instance, refer also to -[Geo with external PostgreSQL instances](external_database.md). - -### Step 4: Configure the frontend application servers on the **secondary** node - -In the architecture overview, there are two machines running the GitLab -application services. These services are enabled selectively in the -configuration. - -Configure the application servers following -[Configuring GitLab for multiple servers](../../high_availability/gitlab.md), then make the -following modifications: - -1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** - cluster, and add the following: - - ```ruby - ## - ## Enable the Geo secondary role - ## - roles ['geo_secondary_role', 'application_role'] - - ## - ## The unique identifier for the Geo node. - ## - gitlab_rails['geo_node_name'] = '' - - ## - ## Disable automatic migrations - ## - gitlab_rails['auto_migrate'] = false - - ## - ## Configure the connection to the tracking DB. And disable application - ## servers from running tracking databases. - ## - geo_secondary['db_host'] = '' - geo_secondary['db_password'] = '' - geo_postgresql['enable'] = false - - ## - ## Configure connection to the streaming replica database, if you haven't - ## already - ## - gitlab_rails['db_host'] = '' - gitlab_rails['db_password'] = '' - - ## - ## Configure connection to Redis, if you haven't already - ## - gitlab_rails['redis_host'] = '' - gitlab_rails['redis_password'] = '' - - ## - ## If you are using custom users not managed by Omnibus, you need to specify - ## UIDs and GIDs like below, and ensure they match between servers in a - ## cluster to avoid permissions issues - ## - user['uid'] = 9000 - user['gid'] = 9000 - web_server['uid'] = 9001 - web_server['gid'] = 9001 - registry['uid'] = 9002 - registry['gid'] = 9002 - ``` - -NOTE: **Note:** -If you had set up PostgreSQL cluster using the omnibus package and you had set -up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in -mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` -mentioned above contains the plaintext passwords. This is used to let the Rails -servers connect to the databases. - -NOTE: **Note:** -Make sure that current node IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of your remote database. - -After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. - -On the secondary the following GitLab frontend services will be enabled: - -- `geo-logcursor` -- `gitlab-pages` -- `gitlab-workhorse` -- `logrotate` -- `nginx` -- `registry` -- `remote-syslog` -- `sidekiq` -- `puma` - -Verify these services by running `sudo gitlab-ctl status` on the frontend -application servers. - -### Step 5: Set up the LoadBalancer for the **secondary** node - -In this topology, a load balancer is required at each geographic location to -route traffic to the application servers. - -See [Load Balancer for GitLab with multiple servers](../../high_availability/load_balancer.md) for -more information. - -### Step 6: Configure the backend application servers on the **secondary** node - -The minimal reference architecture diagram above shows all application services -running together on the same machines. However, for multiple servers we -[strongly recommend running all services separately](../../reference_architectures/index.md). - -For example, a Sidekiq server could be configured similarly to the frontend -application servers above, with some changes to run only the `sidekiq` service: - -1. Edit `/etc/gitlab/gitlab.rb` on each Sidekiq server in the **secondary** - cluster, and add the following: - - ```ruby - ## - ## Enable the Geo secondary role - ## - roles ['geo_secondary_role'] - - ## - ## Enable the Sidekiq service - ## - sidekiq['enable'] = true - - ## - ## Ensure unnecessary services are disabled - ## - alertmanager['enable'] = false - consul['enable'] = false - geo_logcursor['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - postgresql['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - repmgr['enable'] = false - puma['enable'] = false - - ## - ## The unique identifier for the Geo node. - ## - gitlab_rails['geo_node_name'] = '' - - ## - ## Disable automatic migrations - ## - gitlab_rails['auto_migrate'] = false - - ## - ## Configure the connection to the tracking DB. And disable application - ## servers from running tracking databases. - ## - geo_secondary['db_host'] = '' - geo_secondary['db_password'] = '' - geo_postgresql['enable'] = false - - ## - ## Configure connection to the streaming replica database, if you haven't - ## already - ## - gitlab_rails['db_host'] = '' - gitlab_rails['db_password'] = '' - - ## - ## Configure connection to Redis, if you haven't already - ## - gitlab_rails['redis_host'] = '' - gitlab_rails['redis_password'] = '' - - ## - ## If you are using custom users not managed by Omnibus, you need to specify - ## UIDs and GIDs like below, and ensure they match between servers in a - ## cluster to avoid permissions issues - ## - user['uid'] = 9000 - user['gid'] = 9000 - web_server['uid'] = 9001 - web_server['gid'] = 9001 - registry['uid'] = 9002 - registry['gid'] = 9002 - ``` - - You can similarly configure a server to run only the `geo-logcursor` service - with `geo_logcursor['enable'] = true` and disabling Sidekiq with - `sidekiq['enable'] = false`. - - These servers do not need to be attached to the load balancer. +This document was moved to [another location](multiple_servers.md). diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index ac79ca25871..87bd7b69515 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -208,7 +208,7 @@ For information on how to update your Geo nodes to the latest GitLab version, se ### Configuring Geo for multiple servers -For information on configuring Geo for multiple servers, see [Geo for multiple servers](high_availability.md). +For information on configuring Geo for multiple servers, see [Geo for multiple servers](multiple_servers.md). ### Configuring Geo with Object Storage diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md new file mode 100644 index 00000000000..9322c4cc417 --- /dev/null +++ b/doc/administration/geo/replication/multiple_servers.md @@ -0,0 +1,459 @@ +# Geo for multiple servers **(PREMIUM ONLY)** + +This document describes a minimal reference architecture for running Geo +in a multi-server configuration. If your multi-server setup differs from the one +described, it is possible to adapt these instructions to your needs. + +## Architecture overview + +![Geo multi-server diagram](../../high_availability/img/geo-ha-diagram.png) + +_[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_ + +The topology above assumes that the **primary** and **secondary** Geo clusters +are located in two separate locations, on their own virtual network +with private IP addresses. The network is configured such that all machines within +one geographic location can communicate with each other using their private IP addresses. +The IP addresses given are examples and may be different depending on the +network topology of your deployment. + +The only external way to access the two Geo deployments is by HTTPS at +`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. + +NOTE: **Note:** +The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS. + +## Redis and PostgreSQL for multiple servers + +Geo supports: + +- Redis and PostgreSQL on the **primary** node configured for multiple servers. +- Redis on **secondary** nodes configured for multiple servers. + +NOTE: **Note:** +Support for PostgreSQL on **secondary** nodes in multi-server configuration +[is planned](https://gitlab.com/groups/gitlab-org/-/epics/2536). + +Because of the additional complexity involved in setting up this configuration +for PostgreSQL and Redis, it is not covered by this Geo multi-server documentation. + +For more information about setting up a multi-server PostgreSQL cluster and Redis cluster using the omnibus package see the multi-server documentation for +[PostgreSQL](../../high_availability/database.md) and +[Redis](../../high_availability/redis.md), respectively. + +NOTE: **Note:** +It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. + +## Prerequisites: Two working GitLab multi-server clusters + +One cluster will serve as the **primary** node. Use the +[GitLab multi-server documentation](../../reference_architectures/index.md) to set this up. If +you already have a working GitLab instance that is in-use, it can be used as a +**primary**. + +The second cluster will serve as the **secondary** node. Again, use the +[GitLab multi-server documentation](../../reference_architectures/index.md) to set this up. +It's a good idea to log in and test it, however, note that its data will be +wiped out as part of the process of replicating from the **primary**. + +## Configure the GitLab cluster to be the **primary** node + +The following steps enable a GitLab cluster to serve as the **primary** node. + +### Step 1: Configure the **primary** frontend servers + +1. Edit `/etc/gitlab/gitlab.rb` and add the following: + + ```ruby + ## + ## Enable the Geo primary role + ## + roles ['geo_primary_role'] + + ## + ## The unique identifier for the Geo node. + ## + gitlab_rails['geo_node_name'] = '' + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + ``` + +After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. + +NOTE: **Note:** PostgreSQL and Redis should have already been disabled on the +application servers, and connections from the application servers to those +services on the backend servers configured, during normal GitLab multi-server set up. See +multi-server configuration documentation for +[PostgreSQL](../../high_availability/database.md#configuring-the-application-nodes) +and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitlab-application). + +### Step 2: Configure the **primary** database + +1. Edit `/etc/gitlab/gitlab.rb` and add the following: + + ```ruby + ## + ## Configure the Geo primary role and the PostgreSQL role + ## + roles ['geo_primary_role', 'postgres_role'] + ``` + +## Configure a **secondary** node + +A **secondary** cluster is similar to any other GitLab multi-server cluster, with two +major differences: + +- The main PostgreSQL database is a read-only replica of the **primary** node's + PostgreSQL database. +- There is also a single PostgreSQL database for the **secondary** cluster, + called the "tracking database", which tracks the synchronization state of + various resources. + +Therefore, we will set up the multi-server components one-by-one, and include deviations +from the normal multi-server setup. However, we highly recommend first configuring a +brand-new cluster as if it were not part of a Geo setup so that it can be +tested and verified as a working cluster. And only then should it be modified +for use as a Geo **secondary**. This helps to separate problems that are related +and are not related to Geo setup. + +### Step 1: Configure the Redis and Gitaly services on the **secondary** node + +Configure the following services, again using the non-Geo multi-server +documentation: + +- [Configuring Redis for GitLab](../../high_availability/redis.md) for multiple servers. +- [Gitaly](../../high_availability/gitaly.md), which will store data that is + synchronized from the **primary** node. + +NOTE: **Note:** +[NFS](../../high_availability/nfs.md) can be used in place of Gitaly but is not +recommended. + +### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node + +NOTE: **Note:** The following documentation assumes the database will be run on +a single node only. Multi-server PostgreSQL on **secondary** nodes is +[not currently supported](https://gitlab.com/groups/gitlab-org/-/epics/2536). + +Configure the [**secondary** database](database.md) as a read-only replica of +the **primary** database. Use the following as a guide. + +1. Generate an MD5 hash of the desired password for the database user that the + GitLab application will use to access the read-replica database: + + Note that the username (`gitlab` by default) is incorporated into the hash. + + ```shell + gitlab-ctl pg-password-md5 gitlab + # Enter password: + # Confirm password: + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Use this hash to fill in `` in the next step. + +1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the + following: + + ```ruby + ## + ## Configure the Geo secondary role and the PostgreSQL role + ## + roles ['geo_secondary_role', 'postgres_role'] + + ## + ## Secondary address + ## - replace '' with the public or VPC address of your Geo secondary node + ## - replace '' with the public or VPC address of your Geo tracking database node + ## + postgresql['listen_address'] = '' + postgresql['md5_auth_cidr_addresses'] = ['/32', '/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '' + gitlab_rails['db_password'] = '' + + ## + ## When running the Geo tracking database on a separate machine, disable it + ## here and allow connections from the tracking database host. And ensure + ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. + ## + geo_postgresql['enable'] = false + + ## + ## Disable `geo_logcursor` service so Rails doesn't get configured here + ## + geo_logcursor['enable'] = false + ``` + +After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. + +If using an external PostgreSQL instance, refer also to +[Geo with external PostgreSQL instances](external_database.md). + +### Step 3: Configure the tracking database on the **secondary** node + +NOTE: **Note:** This documentation assumes the tracking database will be run on +only a single machine, rather than as a PostgreSQL cluster. + +Configure the tracking database. + +1. Generate an MD5 hash of the desired password for the database user that the + GitLab application will use to access the tracking database: + + Note that the username (`gitlab_geo` by default) is incorporated into the + hash. + + ```shell + gitlab-ctl pg-password-md5 gitlab_geo + # Enter password: + # Confirm password: + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Use this hash to fill in `` in the next + step. + +1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the + following: + + ```ruby + ## + ## Enable the Geo secondary tracking database + ## + geo_postgresql['enable'] = true + geo_postgresql['listen_address'] = '' + geo_postgresql['sql_user_password'] = '' + + ## + ## Configure FDW connection to the replica database + ## + geo_secondary['db_fdw'] = true + geo_postgresql['fdw_external_password'] = '' + geo_postgresql['md5_auth_cidr_addresses'] = ['/32'] + gitlab_rails['db_host'] = '' + + # Prevent reconfigure from attempting to run migrations on the replica DB + gitlab_rails['auto_migrate'] = false + + ## + ## Disable all other services that aren't needed, since we don't have a role + ## that does this. + ## + alertmanager['enable'] = false + consul['enable'] = false + gitaly['enable'] = false + gitlab_exporter['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + node_exporter['enable'] = false + pgbouncer_exporter['enable'] = false + postgresql['enable'] = false + prometheus['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + repmgr['enable'] = false + sidekiq['enable'] = false + puma['enable'] = false + ``` + +After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. + +If using an external PostgreSQL instance, refer also to +[Geo with external PostgreSQL instances](external_database.md). + +### Step 4: Configure the frontend application servers on the **secondary** node + +In the architecture overview, there are two machines running the GitLab +application services. These services are enabled selectively in the +configuration. + +Configure the application servers following +[Configuring GitLab for multiple servers](../../high_availability/gitlab.md), then make the +following modifications: + +1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** + cluster, and add the following: + + ```ruby + ## + ## Enable the Geo secondary role + ## + roles ['geo_secondary_role', 'application_role'] + + ## + ## The unique identifier for the Geo node. + ## + gitlab_rails['geo_node_name'] = '' + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + + ## + ## Configure the connection to the tracking DB. And disable application + ## servers from running tracking databases. + ## + geo_secondary['db_host'] = '' + geo_secondary['db_password'] = '' + geo_postgresql['enable'] = false + + ## + ## Configure connection to the streaming replica database, if you haven't + ## already + ## + gitlab_rails['db_host'] = '' + gitlab_rails['db_password'] = '' + + ## + ## Configure connection to Redis, if you haven't already + ## + gitlab_rails['redis_host'] = '' + gitlab_rails['redis_password'] = '' + + ## + ## If you are using custom users not managed by Omnibus, you need to specify + ## UIDs and GIDs like below, and ensure they match between servers in a + ## cluster to avoid permissions issues + ## + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` + +NOTE: **Note:** +If you had set up PostgreSQL cluster using the omnibus package and you had set +up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in +mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` +mentioned above contains the plaintext passwords. This is used to let the Rails +servers connect to the databases. + +NOTE: **Note:** +Make sure that current node IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of your remote database. + +After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. + +On the secondary the following GitLab frontend services will be enabled: + +- `geo-logcursor` +- `gitlab-pages` +- `gitlab-workhorse` +- `logrotate` +- `nginx` +- `registry` +- `remote-syslog` +- `sidekiq` +- `puma` + +Verify these services by running `sudo gitlab-ctl status` on the frontend +application servers. + +### Step 5: Set up the LoadBalancer for the **secondary** node + +In this topology, a load balancer is required at each geographic location to +route traffic to the application servers. + +See [Load Balancer for GitLab with multiple servers](../../high_availability/load_balancer.md) for +more information. + +### Step 6: Configure the backend application servers on the **secondary** node + +The minimal reference architecture diagram above shows all application services +running together on the same machines. However, for multiple servers we +[strongly recommend running all services separately](../../reference_architectures/index.md). + +For example, a Sidekiq server could be configured similarly to the frontend +application servers above, with some changes to run only the `sidekiq` service: + +1. Edit `/etc/gitlab/gitlab.rb` on each Sidekiq server in the **secondary** + cluster, and add the following: + + ```ruby + ## + ## Enable the Geo secondary role + ## + roles ['geo_secondary_role'] + + ## + ## Enable the Sidekiq service + ## + sidekiq['enable'] = true + + ## + ## Ensure unnecessary services are disabled + ## + alertmanager['enable'] = false + consul['enable'] = false + geo_logcursor['enable'] = false + gitaly['enable'] = false + gitlab_exporter['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + node_exporter['enable'] = false + pgbouncer_exporter['enable'] = false + postgresql['enable'] = false + prometheus['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + repmgr['enable'] = false + puma['enable'] = false + + ## + ## The unique identifier for the Geo node. + ## + gitlab_rails['geo_node_name'] = '' + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + + ## + ## Configure the connection to the tracking DB. And disable application + ## servers from running tracking databases. + ## + geo_secondary['db_host'] = '' + geo_secondary['db_password'] = '' + geo_postgresql['enable'] = false + + ## + ## Configure connection to the streaming replica database, if you haven't + ## already + ## + gitlab_rails['db_host'] = '' + gitlab_rails['db_password'] = '' + + ## + ## Configure connection to Redis, if you haven't already + ## + gitlab_rails['redis_host'] = '' + gitlab_rails['redis_password'] = '' + + ## + ## If you are using custom users not managed by Omnibus, you need to specify + ## UIDs and GIDs like below, and ensure they match between servers in a + ## cluster to avoid permissions issues + ## + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` + + You can similarly configure a server to run only the `geo-logcursor` service + with `geo_logcursor['enable'] = true` and disabling Sidekiq with + `sidekiq['enable'] = false`. + + These servers do not need to be attached to the load balancer. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 36adb12d63d..3adfc83ce06 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -651,9 +651,9 @@ To check the configuration: ``` This password is normally set on the tracking database during - [Step 3: Configure the tracking database on the secondary node](high_availability.md#step-3-configure-the-tracking-database-on-the-secondary-node), + [Step 3: Configure the tracking database on the secondary node](multiple_servers.md#step-3-configure-the-tracking-database-on-the-secondary-node), and it is set on the app nodes during - [Step 4: Configure the frontend application servers on the secondary node](high_availability.md#step-4-configure-the-frontend-application-servers-on-the-secondary-node). + [Step 4: Configure the frontend application servers on the secondary node](multiple_servers.md#step-4-configure-the-frontend-application-servers-on-the-secondary-node). 1. Check whether any tables are present with the following statement: diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md index 553e8fe6932..2adf06a8e95 100644 --- a/doc/api/api_resources.md +++ b/doc/api/api_resources.md @@ -26,6 +26,7 @@ The following API resources are available in the project context: | [Custom attributes](custom_attributes.md) | `/projects/:id/custom_attributes` (also available for groups and users) | | [Dependencies](dependencies.md) **(ULTIMATE)** | `/projects/:id/dependencies` | | [Deploy keys](deploy_keys.md) | `/projects/:id/deploy_keys` (also available standalone) | +| [Freeze Periods](freeze_periods.md) | `/projects/:id/freeze_periods` | | [Deployments](deployments.md) | `/projects/:id/deployments` | | [Discussions](discussions.md) (threaded comments) | `/projects/:id/issues/.../discussions`, `/projects/:id/snippets/.../discussions`, `/projects/:id/merge_requests/.../discussions`, `/projects/:id/commits/.../discussions` (also available for groups) | | [Environments](environments.md) | `/projects/:id/environments` | diff --git a/doc/api/freeze_periods.md b/doc/api/freeze_periods.md new file mode 100644 index 00000000000..d5f165015e7 --- /dev/null +++ b/doc/api/freeze_periods.md @@ -0,0 +1,168 @@ +# Freeze Periods API + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29382) in GitLab 13.0. + +You can use the Freeze Periods API to manipulate GitLab's [Freeze Period](../user/project/releases/index.md#set-a-deploy-freeze) entries. + +## Permissions and security + +Only users with Maintainer [permissions](../user/permissions.md) can +interact with the Freeze Period API endpoints. + +## List Freeze Periods + +Paginated list of Freeze Periods, sorted by `created_at`. + +```plaintext +GET /projects/:id/freeze_periods +``` + +| Attribute | Type | Required | Description | +| ------------- | -------------- | -------- | ----------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: gVWYVHDRzXiRpN1rUC8T" "https://gitlab.example.com/api/v4/projects/19/freeze_periods" +``` + +Example response: + +```json +[ + { + "id":1, + "freeze_start":"0 23 * * 5", + "freeze_end":"0 8 * * 1", + "cron_timezone":"UTC", + "created_at":"2020-05-15T17:03:35.702Z", + "updated_at":"2020-05-15T17:06:41.566Z" + } +] +``` + +## Get a Freeze Period by a `freeze_period_id` + +Get a Freeze Period for the given `freeze_period_id`. + +```plaintext +GET /projects/:id/freeze_periods/:freeze_period_id +``` + +| Attribute | Type | Required | Description | +| ------------- | -------------- | -------- | ----------------------------------------------------------------------------------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `freeze_period_id` | string | yes | The database ID of the Freeze Period. | + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: gVWYVHDRzXiRpN1rUC8T" "https://gitlab.example.com/api/v4/projects/19/freeze_periods/1" +``` + +Example response: + +```json +{ + "id":1, + "freeze_start":"0 23 * * 5", + "freeze_end":"0 8 * * 1", + "cron_timezone":"UTC", + "created_at":"2020-05-15T17:03:35.702Z", + "updated_at":"2020-05-15T17:06:41.566Z" +} +``` + +## Create a Freeze Period + +Create a Freeze Period. + +```plaintext +POST /projects/:id/freeze_periods +``` + +| Attribute | Type | Required | Description | +| -------------------| --------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `freeze_start` | string | yes | Start of the Freeze Period in [cron](https://crontab.guru/) format. | +| `freeze_end` | string | yes | End of the Freeze Period in [cron](https://crontab.guru/) format. | +| `cron_timezone` | string | no | The timezone for the cron fields, defaults to UTC if not provided. | + +Example request: + +```shell +curl --header 'Content-Type: application/json' --header "PRIVATE-TOKEN: gVWYVHDRzXiRpN1rUC8T" \ + --data '{ "freeze_start": "0 23 * * 5", "freeze_end": "0 7 * * 1", "cron_timezone": "UTC" }' \ + --request POST https://gitlab.example.com/api/v4/projects/19/freeze_periods +``` + +Example response: + +```json +{ + "id":1, + "freeze_start":"0 23 * * 5", + "freeze_end":"0 7 * * 1", + "cron_timezone":"UTC", + "created_at":"2020-05-15T17:03:35.702Z", + "updated_at":"2020-05-15T17:03:35.702Z" +} +``` + +## Update a Freeze Period + +Update a Freeze Period for the given `freeze_period_id`. + +```plaintext +PUT /projects/:id/freeze_periods/:tag_name +``` + +| Attribute | Type | Required | Description | +| ------------- | --------------- | -------- | ----------------------------------------------------------------------------------------------------------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `freeze_period_id` | integer or string | yes | The database ID of the Freeze Period. | +| `freeze_start` | string | no | Start of the Freeze Period in [cron](https://crontab.guru/) format. | +| `freeze_end` | string | no | End of the Freeze Period in [cron](https://crontab.guru/) format. | +| `cron_timezone` | string | no | The timezone for the cron fields. | + +Example request: + +```shell +curl --header 'Content-Type: application/json' --header "PRIVATE-TOKEN: gVWYVHDRzXiRpN1rUC8T" \ + --data '{ "freeze_end": "0 8 * * 1" }' \ + --request PUT https://gitlab.example.com/api/v4/projects/19/freeze_periods/1 +``` + +Example response: + +```json +{ + "id":1, + "freeze_start":"0 23 * * 5", + "freeze_end":"0 8 * * 1", + "cron_timezone":"UTC", + "created_at":"2020-05-15T17:03:35.702Z", + "updated_at":"2020-05-15T17:06:41.566Z" +} +``` + +## Delete a Freeze Period + +Delete a Freeze Period for the given `freeze_period_id`. + +```plaintext +DELETE /projects/:id/freeze_periods/:freeze_period_id +``` + +| Attribute | Type | Required | Description | +| ------------- | -------------- | -------- | ----------------------------------------------------------------------------------- | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `freeze_period_id` | string | yes | The database ID of the Freeze Period. | + +Example request: + +```shell +curl --request DELETE --header "PRIVATE-TOKEN: gVWYVHDRzXiRpN1rUC8T" "https://gitlab.example.com/api/v4/projects/19/freeze_periods/1" + +``` diff --git a/doc/gitlab-geo/ha.md b/doc/gitlab-geo/ha.md index 23ed11eaf09..0be70791d45 100644 --- a/doc/gitlab-geo/ha.md +++ b/doc/gitlab-geo/ha.md @@ -1,5 +1,5 @@ --- -redirect_to: '../administration/geo/replication/high_availability.md' +redirect_to: '../administration/geo/replication/multiple_servers.md' --- -This document was moved to [another location](../administration/geo/replication/high_availability.md). +This document was moved to [another location](../administration/geo/replication/multiple_servers.md). diff --git a/doc/user/project/merge_requests/accessibility_testing.md b/doc/user/project/merge_requests/accessibility_testing.md index eaf950870cb..427761281f6 100644 --- a/doc/user/project/merge_requests/accessibility_testing.md +++ b/doc/user/project/merge_requests/accessibility_testing.md @@ -18,11 +18,23 @@ measuring the accessibility of web sites, and has built a simple This job outputs accessibility violations, warnings, and notices for each page analyzed to a file called `accessibility`. +## Accessibility Merge Request widget + [Since GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/39425), in addition to the report artifact that is created, GitLab will also show the Accessibility Report in the merge request widget area: ![Accessibility Merge Request Widget](img/accessibility_mr_widget_v13_0.png) +This widget comes with the `:accessibility_report_view` feature flag disabled by default while we test feature stability. +Once we have determined the widget is stable, this feature will be enabled by default. + +To enable this feature, ask a GitLab administrator with [Rails console access](../../../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the +following command: + +```ruby +Feature.enable(:accessibility_report_view) +``` + ## Configure Accessibility Testing This example shows how to run [pa11y](https://pa11y.org/) diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index df4ca28d081..8ea51514515 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -354,6 +354,39 @@ terminal. Read the [GitLab Releaser documentation](https://gitlab.com/gitlab-org/gitlab-releaser/-/tree/master/docs/index.md) for details. +## Set a deploy freeze + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29382) in GitLab 13.0. + +With a deploy freeze, you can prevent an unintended production release during a +period of time you specify, whether a company event or public holiday. You can +now rely on the enforcement of policies that are typically outside the scope of +GitLab to reduce uncertainty and risk when automating deployments. + +Deploy freeze periods are set at the Project level, and may be created and +managed using the [Freeze Periods API](../../../api/freeze_periods.md). +Each Freeze Period has a `freeze_start` and a `freeze_end`, which are defined +as [crontab](https://crontab.guru/) entries. If a project contains multiple +freeze periods, all will apply, and should they overlap, the freeze covers the +complete overlapped period. + +During pipeline processing, GitLab CI creates an environment variable named +`$CI_ENVIRONMENT_FROZEN` if the currently executing job is within a +Freeze Period. + +To take advantage of this variable, create a `rules` entry in your +`gitlab-ci.yaml` to prevent the deployment job from executing. + +For example: + +```yaml +deploy_to_production: + stage: deploy + script: deploy_to_prod.sh + rules: + - if: $CI_ENVIRONMENT_FROZEN == null +``` +