From 95dc24081cbfcdff15426b0b558d3456d0bf9fed Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Wed, 1 Dec 2021 03:13:29 +0000 Subject: [PATCH] Add latest changes from gitlab-org/gitlab@master --- doc/administration/gitaly/praefect.md | 84 +++++++++++ doc/administration/gitaly/troubleshooting.md | 24 +-- doc/ci/pipelines/job_artifacts.md | 63 +++++++- doc/ci/yaml/index.md | 146 ++++++++----------- doc/user/gitlab_com/index.md | 17 +++ 5 files changed, 242 insertions(+), 92 deletions(-) diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index c39ea2e6b8c..fbc61855b76 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1332,6 +1332,90 @@ To enable writes again in GitLab 13.0 to 14.0, an administrator can: [accept data loss](#enable-writes-or-accept-data-loss) if necessary, depending on the version of GitLab. +## Retrieve repository metadata + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3481) in GitLab 14.6. + +Gitaly Cluster maintains a [metadata database](index.md#components) about the repositories stored on the cluster. Use the `praefect metadata` subcommand +to inspect the metadata for troubleshooting. + +You can retrieve a repository's metadata by its Praefect-assigned repository ID: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id +``` + +You can also retrieve a repository's metadata by its virtual storage and relative path: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -virtual-storage -relative-path +``` + +### Examples + +To retrieve the metadata for a repository with a Praefect-assigned repository ID of 1: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id 1 +``` + +To retrieve the metadata for a repository with virtual storage `default` and relative path `@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git`: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -virtual-storage default -relative-path @hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git +``` + +Either of these examples retrieve the following metadata for an example repository: + +```plaintext +Repository ID: 54771 +Virtual Storage: "default" +Relative Path: "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git" +Replica Path: "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git" +Primary: "gitaly-1" +Generation: 1 +Replicas: +- Storage: "gitaly-1" + Assigned: true + Generation: 1, fully up to date + Healthy: true + Valid Primary: true +- Storage: "gitaly-2" + Assigned: true + Generation: 0, behind by 1 changes + Healthy: true + Valid Primary: false +- Storage: "gitaly-3" + Assigned: true + Generation: replica not yet created + Healthy: false + Valid Primary: false +``` + +### Available metadata + +The metadata retrieved by `praefect metadata` includes the fields in the following tables. + +| Field | Description | +|:------------------|:-------------------------------------------------------------------------------------------------------------------| +| `Repository ID` | Permanent unique ID assigned to the repository by Praefect. Different to the ID GitLab uses for repositories. | +| `Virtual Storage` | Name of the virtual storage the repository is stored in. | +| `Relative Path` | Repository's path in the virtual storage. | +| `Replica Path` | Where on the Gitaly node's disk the repository's replicas are stored. | +| `Primary` | Current primary of the repository. | +| `Generation` | Used by Praefect to track repository changes. Each write in the repository increments the repository's generation. | +| `Replicas` | A list of replicas that exist or are expected to exist. | + +For each replica, the following metadata is available: + +| `Replicas` Field | Description | +|:-----------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `Storage` | Name of the Gitaly storage that contains the replica. | +| `Assigned` | Indicates whether the replica is expected to exist in the storage. Can be `false` if a Gitaly node is removed from the cluster or if the storage contains an extra copy after the repository's replication factor was decreased. | +| `Generation` | Latest confirmed generation of the replica. It indicates:

- The replica is fully up to date if the generation matches the repository's generation.
- The replica is outdated if the replica's generation is less than the repository's generation.
- `replica not yet created` if the replica does not yet exist at all on the storage. | +| `Healthy` | Indicates whether the Gitaly node that is hosting this replica is considered healthy by the consensus of Praefect nodes. | +| `Valid Primary` | Indicates whether the replica is fit to serve as the primary node. If the repository's primary is not a valid primary, a failover occurs on the next write to the repository if there is another replica that is a valid primary. A replica is a valid primary if:

- It is stored on a healthy Gitaly node.
- It is fully up to date.
- It is not targeted by a pending deletion job from decreasing replication factor.
- It is assigned. | + ## Unavailable repositories > - From GitLab 13.0 through 14.0, repositories became read-only if they were outdated on the primary but fully up to date on a healthy secondary. `dataloss` sub-command displays read-only repositories by default through these versions. diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index d31ed18d62e..2e5f571d694 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -374,17 +374,23 @@ Here are common errors and potential causes: ### Determine primary Gitaly node -To determine the current primary Gitaly node for a specific Praefect node: +To determine the primary node of a repository: -- Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the - [`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json). - This is recommended. -- If you do not have Grafana set up, use the following command on each host of each - Praefect node: +- In GitLab 14.6 and later, use the [`praefect metadata`](praefect.md#retrieve-repository-metadata) subcommand. +- In GitLab 13.12 to GitLab 14.5 with [repository-specific primaries](praefect.md#repository-specific-primary-nodes), + use the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums). +- With legacy election strategies in GitLab 13.12 and earlier, the primary was the same for all repositories in a virtual storage. + To determine the current primary Gitaly node for a specific virtual storage: - ```shell - curl localhost:9652/metrics | grep gitaly_praefect_primaries` - ``` + - Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the + [`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json). + This is recommended. + - If you do not have Grafana set up, use the following command on each host of each + Praefect node: + + ```shell + curl localhost:9652/metrics | grep gitaly_praefect_primaries` + ``` ### Check that repositories are in sync diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index 783f8fc6b8d..73d9c1f728c 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -48,7 +48,45 @@ is used. If you run two types of pipelines (like branch and scheduled) for the same ref, the pipeline that finishes later creates the job artifact. -For more examples, view the [keyword reference for the `.gitlab-ci.yml` file](../yaml/index.md#artifacts). +To disable artifact passing, define the job with empty [dependencies](../yaml/index.md#dependencies): + +```yaml +job: + stage: build + script: make build + dependencies: [] +``` + +You may want to create artifacts only for tagged releases to avoid filling the +build server storage with temporary build artifacts. For example, use [`rules`](../yaml/index.md#rules) +to create artifacts only for tags: + +```yaml +default-job: + script: + - mvn test -U + rules: + - if: $CI_COMMIT_BRANCH + +release-job: + script: + - mvn package -U + artifacts: + paths: + - target/*.war + rules: + - if: $CI_COMMIT_TAG +``` + +You can use wildcards for directories too. For example, if you want to get all the +files inside the directories that end with `xyz`: + +```yaml +job: + artifacts: + paths: + - path/*xyz/* +``` ### Use CI/CD variables to define the artifacts name @@ -153,6 +191,29 @@ artifacts: - binaries/temp/**/* ``` +### Add untracked files to artifacts + +Use [`artifacts:untracked`](../yaml/index.md#artifactsuntracked) to add all Git untracked +files as artifacts (along with the paths defined in [`artifacts:paths`](../yaml/index.md#artifactspaths)). + +Save all Git untracked files and files in `binaries`: + +```yaml +artifacts: + untracked: true + paths: + - binaries/ +``` + +Save all untracked files but [exclude](../yaml/index.md#artifactsexclude) `*.txt`: + +```yaml +artifacts: + untracked: true + exclude: + - "*.txt" +``` + ## Download job artifacts You can download job artifacts or view the job archive: diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index 7bb861f1787..6a7c9ae1673 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -819,64 +819,42 @@ job: #### `artifacts:paths` Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly -link outside it. You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) -patterns and: +link outside it. -- In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), - [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). -- In GitLab Runner 12.10 and earlier, [`filepath.Match`](https://pkg.go.dev/path/filepath#Match). +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). -To restrict which jobs a specific job fetches artifacts from, see [dependencies](#dependencies). +**Possible inputs**: -Send all files in `binaries` and `.config`: +- An array of file paths, relative to the project directory. +- You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) + patterns and: + - In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), + [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). + - In GitLab Runner 12.10 and earlier, [`filepath.Match`](https://pkg.go.dev/path/filepath#Match). -```yaml -artifacts: - paths: - - binaries/ - - .config -``` - -To disable artifact passing, define the job with empty [dependencies](#dependencies): - -```yaml -job: - stage: build - script: make build - dependencies: [] -``` - -You may want to create artifacts only for tagged releases to avoid filling the -build server storage with temporary build artifacts. - -Create artifacts only for tags (`default-job` doesn't create artifacts): - -```yaml -default-job: - script: - - mvn test -U - rules: - - if: $CI_COMMIT_BRANCH - -release-job: - script: - - mvn package -U - artifacts: - paths: - - target/*.war - rules: - - if: $CI_COMMIT_TAG -``` - -You can use wildcards for directories too. For example, if you want to get all the files inside the directories that end with `xyz`: +**Example of `artifacts:paths`**: ```yaml job: artifacts: paths: - - path/*xyz/* + - binaries/ + - .config ``` +This example creates an artifact with `.config` and all the files in the `binaries` directory. + +**Additional details**: + +- If not used with [`artifacts:name`](#artifactsname) defined, the artifacts file + is named `artifacts`, which becomes `artifacts.zip` when downloaded. + +**Related topics**: + +- To restrict which jobs a specific job fetches artifacts from, see [`dependencies`](#dependencies). +- [Create job artifacts](../pipelines/job_artifacts.md#create-job-artifacts). + #### `artifacts:public` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 @@ -887,20 +865,25 @@ job: Use `artifacts:public` to determine whether the job artifacts should be publicly available. -The default for `artifacts:public` is `true` which means that the artifacts in -public pipelines are available for download by anonymous and guest users: - -```yaml -artifacts: - public: true -``` +When `artifacts:public` is `true` (default), the artifacts in +public pipelines are available for download by anonymous and guest users. To deny read access for anonymous and guest users to artifacts in public pipelines, set `artifacts:public` to `false`: +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). + +**Possible inputs**: + +- `true` (default if not defined) or `false`. + +**Example of `artifacts:paths`**: + ```yaml -artifacts: - public: false +job: + artifacts: + public: false ``` #### `artifacts:reports` @@ -941,45 +924,44 @@ Use `artifacts:untracked` to add all Git untracked files as artifacts (along with the paths defined in `artifacts:paths`). `artifacts:untracked` ignores configuration in the repository's `.gitignore` file. -Send all Git untracked files: +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). + +**Possible inputs**: + +- `true` or `false` (default if not defined). + +**Example of `artifacts:untracked`**: + +Save all Git untracked files: ```yaml -artifacts: - untracked: true +job: + artifacts: + untracked: true ``` -Send all Git untracked files and files in `binaries`: +**Related topics**: -```yaml -artifacts: - untracked: true - paths: - - binaries/ -``` - -Send all untracked files but [exclude](#artifactsexclude) `*.txt`: - -```yaml -artifacts: - untracked: true - exclude: - - "*.txt" -``` +- [Add untracked files to artifacts](../pipelines/job_artifacts.md#add-untracked-files-to-artifacts). #### `artifacts:when` Use `artifacts:when` to upload artifacts on job failure or despite the failure. -`artifacts:when` can be set to one of the following values: +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). -1. `on_success` (default): Upload artifacts only when the job succeeds. -1. `on_failure`: Upload artifacts only when the job fails. -1. `always`: Always upload artifacts. For example, when - [uploading artifacts](../unit_test_reports.md#viewing-junit-screenshots-on-gitlab) required to - troubleshoot failing tests. +**Possible inputs**: -For example, to upload artifacts only when a job fails: +- `on_success` (default): Upload artifacts only when the job succeeds. +- `on_failure`: Upload artifacts only when the job fails. +- `always`: Always upload artifacts. For example, when + [uploading artifacts](../unit_test_reports.md#viewing-junit-screenshots-on-gitlab) + required to troubleshoot failing tests. + +**Example of `artifacts:when`**: ```yaml job: diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index 900490053d6..356f53e91eb 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -143,6 +143,23 @@ the related documentation. | [Max registered runners](../../administration/instance_limits.md#number-of-registered-runners-per-scope) | Free tier: `50` per-group / `50` per-project
All paid tiers: `1_000` per-group / `1_000` per-project | `1_000` per-group / `1_000` per-project | | [Limit dotenv variables](../../administration/instance_limits.md#limit-dotenv-variables) | Free tier: `50` / Premium tier: `100` / Ultimate tier: `150` | 150 | +## Package registry limits + +The [maximum file size](../../administration/instance_limits.md#file-size-limits) +for a package uploaded to the [GitLab Package Registry](../../user/packages/package_registry/index.md) +varies by format: + +| Package type | GitLab.com | Default | +|--------------|------------|---------| +| Conan | 5 GB | 3 GB | +| Generic | 5 GB | 5 GB | +| Helm | 5 MB | 5 MB | +| Maven | 5 GB | 3 GB | +| npm: | 5 GB | 500 MB | +| NuGet | 5 GB | 500 MB | +| PyPI | 5 GB | 3 GB | +| Terraform | 1 GB | 1 GB | + ## Account and limit settings GitLab.com has the following [account limits](../admin_area/settings/account_and_limit_settings.md)