diff --git a/.gitlab/ci/docs.gitlab-ci.yml b/.gitlab/ci/docs.gitlab-ci.yml index 6aa3d533315..ae36c0cea70 100644 --- a/.gitlab/ci/docs.gitlab-ci.yml +++ b/.gitlab/ci/docs.gitlab-ci.yml @@ -53,7 +53,7 @@ docs-lint markdown: docs-lint links: extends: - .docs:rules:docs-lint - image: registry.gitlab.com/gitlab-org/gitlab-docs/lint-html:alpine-3.14-ruby-2.7.4-0fae0f62 + image: registry.gitlab.com/gitlab-org/gitlab-docs/lint-html:alpine-3.14-ruby-2.7.5-08847baa stage: lint needs: [] script: diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 3c4fd276187..31316693917 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -295,18 +295,28 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t ### Manually remove repositories -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3767) in GitLab 14.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3767) in GitLab 14.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4054) in GitLab 14.6, support for dry-run mode. -The `remove-repository` Praefect sub-command removes repositories from a Gitaly Cluster. It removes -all state associated with a given repository including: +The `remove-repository` Praefect sub-command removes a repository from a Gitaly Cluster, and all state associated with a given repository including: - On-disk repositories on all relevant Gitaly nodes. - Any database state tracked by Praefect. +In GitLab 14.6 and later, by default, the command operates in dry-run mode. In earlier versions, the command didn't support dry-run mode. For example: + ```shell sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage -repository ``` +- Replace `` with the name of the virtual storage containing the repository. +- Replace `` with the relative path of the repository to remove. +- In GitLab 14.6 and later, add `-apply` to run the command outside of dry-run mode and remove the repository. For example: + + ```shell + sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage -repository -apply + ``` + - `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following: ```ruby diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 5cccd9e3293..fdd281c1a90 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -353,6 +353,69 @@ that do not exist in a repository. The following sections provide possible solutions to Gitaly Cluster errors. +### Check cluster health + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/) in GitLab 14.6. + +The `check` Praefect sub-command runs a series of checks to determine the health of the Gitaly Cluster. + +```shell +gitlab-ctl praefect check +``` + +The following sections describe the checks that are run. + +#### Praefect migrations + +Because Database migrations must be up to date for Praefect to work correctly, checks if Praefect migrations are up to date. + +If this check fails: + +1. See the `schema_migrations` table in the database to see which migrations have run. +1. Run `praefect sql-migrate` to bring the migrations up to date. + +#### Node connectivity and disk access + +Checks if Praefect can reach all of its Gitaly nodes, and if each Gitaly node has read and write access to all of its storages. + +If this check fails: + +1. Confirm the network addresses and tokens are set up correctly: + - In the Praefect configuration. + - In each Gitaly node's configuration. +1. On the Gitaly nodes, check that the `gitaly` process being run as `git`. There might be a permissions issue that is preventing Gitaly from + accessing its storage directories. +1. Confirm that there are no issues with the network that connects Praefect to Gitaly nodes. + +#### Database read and write access + +Checks if Praefect can read from and write to the database. + +If this check fails: + +1. See if the Praefect database is in recovery mode. In recovery mode, tables may be read only. To check, run: + + ```sql + select pg_is_in_recovery() + ``` + +1. Confirm that the user that Praefect uses to connect to PostgreSQL has read and write access to the database. +1. See if the database has been placed into read-only mode. To check, run: + + ```sql + show default_transaction_read_only + ``` + +#### Inaccessible repositories + +Checks how many repositories are inaccessible because they are missing a primary assignment, or their primary is unavailable. + +If this check fails: + +1. See if any Gitaly nodes are down. Run `praefect ping-nodes` to check. +1. Check if there is a high load on the Praefect database. If the Praefect database is slow to respond, it can lead health checks failing to persist + to the database, leading Praefect to think nodes are unhealthy. + ### Praefect errors in logs If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 53584e1ff8b..64b5ddbd165 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -154,37 +154,7 @@ _The artifacts are stored by default in ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local artifacts to the object storage: - - ```shell - gitlab-rake gitlab:artifacts:migrate - ``` - -1. Optional. Verify all files migrated properly. - From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) - (`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: - - ```shell - gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts; - - total | filesystem | objectstg - ------+------------+----------- - 2409 | 0 | 2409 - ``` - - Verify no files on disk in `artifacts` folder: - - ```shell - sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp/cache | wc -l - ``` - - In some cases, you may need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files) - to clean up orphaned artifacts. - -WARNING: -JUnit test report artifact (`junit.xml.gz`) migration -[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991) -by the `gitlab:artifacts:migrate` script. +1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). **In installations from source:** @@ -208,36 +178,7 @@ _The artifacts are stored by default in ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -1. Migrate any existing local artifacts to the object storage: - - ```shell - sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production - ``` - -1. Optional: Verify all files migrated properly. - From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: - - ```shell - gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts; - - total | filesystem | objectstg - ------+------------+----------- - 2409 | 0 | 2409 - ``` - - Verify no files on disk in `artifacts` folder: - - ```shell - sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp/cache | wc -l - ``` - - In some cases, you may need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files) - to clean up orphaned artifacts. - -WARNING: -JUnit test report artifact (`junit.xml.gz`) migration -[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991) -by the `gitlab:artifacts:migrate` script. +1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). ### OpenStack example @@ -267,11 +208,7 @@ _The uploads are stored by default in ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local artifacts to the object storage: - - ```shell - gitlab-rake gitlab:artifacts:migrate - ``` +1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). --- @@ -302,11 +239,55 @@ _The uploads are stored by default in ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -1. Migrate any existing local artifacts to the object storage: +1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). - ```shell - sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production - ``` +### Migrating to object storage + +After [configuring the object storage](#using-object-storage), use the following task to +migrate existing job artifacts from the local storage to the remote storage. +The processing is done in a background worker and requires **no downtime**. + +**In Omnibus installations:** + +```shell +gitlab-rake gitlab:artifacts:migrate +``` + +**In installations from source:** + +```shell +sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production +``` + +You can optionally track progress and verify that all job artifacts migrated successfully using the +[PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): + +- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. + +Verify `objectstg` below (where `store=2`) has count of all job artifacts: + +```shell +gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts; + +total | filesystem | objectstg +------+------------+----------- + 19 | 0 | 19 +``` + +Verify that there are no files on disk in the `artifacts` folder: + +```shell +sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp | wc -l +``` + +In some cases, you need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files) +to clean up orphaned artifacts. + +WARNING: +JUnit test report artifact (`junit.xml.gz`) migration +[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991) +by the `gitlab:artifacts:migrate` Rake task. ### Migrating from object storage to local storage diff --git a/doc/integration/gitpod.md b/doc/integration/gitpod.md index 26520df18fa..977e794396e 100644 --- a/doc/integration/gitpod.md +++ b/doc/integration/gitpod.md @@ -5,7 +5,7 @@ group: Editor info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" --- -# Gitpod Integration **(FREE)** +# Gitpod integration **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/228893) in GitLab 13.4. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/258206) in GitLab 13.8 @@ -38,7 +38,7 @@ With the Gitpod integration enabled for your GitLab instance, to enable it for y 1. In the top-right corner, select your avatar. 1. Select **Preferences**. 1. Under **Preferences**, locate the **Integrations** section. -1. Check the **Enable Gitpod integration** checkbox and select the **Save changes** button. +1. Select the **Enable Gitpod integration** checkbox and select **Save changes**. ## Configure a self-managed instance **(FREE SELF)** @@ -50,9 +50,9 @@ For GitLab self-managed instances, a GitLab administrator needs to: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Gitpod** configuration section. - 1. Check the **Enable Gitpod integration** checkbox. + 1. Select the **Enable Gitpod integration** checkbox. 1. Add your Gitpod instance URL (for example, `https://gitpod.example.com`). - 1. Select the **Save changes** button. + 1. Select **Save changes**. Your users can then [enable it for themselves](#enable-gitpod-in-your-user-settings).