kotovalexarian-likes-gitlab/gitlab-org--gitlab-foss
1
0
Fork 0
Code Releases Activity

Add latest changes from gitlab-org/gitlab@master

Browse source
This commit is contained in:
GitLab Bot 2021-12-01 03:13:29 +00:00
parent 5f844d0ef6
commit 95dc24081c
5 changed files with 242 additions and 92 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

84
doc/administration/gitaly/praefect.md
View file

@ -1332,6 +1332,90 @@ To enable writes again in GitLab 13.0 to 14.0, an administrator can:
[accept data loss](#enable-writes-or-accept-data-loss) if necessary, depending on the version of
GitLab.
## Retrieve repository metadata
> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3481) in GitLab 14.6.
Gitaly Cluster maintains a [metadata database](index.md#components) about the repositories stored on the cluster. Use the `praefect metadata` subcommand
to inspect the metadata for troubleshooting.
You can retrieve a repository's metadata by its Praefect-assigned repository ID:
```shell
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id <repository-id>
```
You can also retrieve a repository's metadata by its virtual storage and relative path:
```shell
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -virtual-storage <virtual-storage> -relative-path <relative-path>
```
### Examples
To retrieve the metadata for a repository with a Praefect-assigned repository ID of 1:
```shell
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id 1
```
To retrieve the metadata for a repository with virtual storage `default` and relative path `@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git`:
```shell
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -virtual-storage default -relative-path @hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git
```
Either of these examples retrieve the following metadata for an example repository:
```plaintext
Repository ID: 54771
Virtual Storage: "default"
Relative Path: "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git"
Replica Path: "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git"
Primary: "gitaly-1"
Generation: 1
Replicas:
- Storage: "gitaly-1"
Assigned: true
Generation: 1, fully up to date
Healthy: true
Valid Primary: true
- Storage: "gitaly-2"
Assigned: true
Generation: 0, behind by 1 changes
Healthy: true
Valid Primary: false
- Storage: "gitaly-3"
Assigned: true
Generation: replica not yet created
Healthy: false
Valid Primary: false
```
### Available metadata
The metadata retrieved by `praefect metadata` includes the fields in the following tables.
| Field | Description |
|:------------------|:-------------------------------------------------------------------------------------------------------------------|
| `Repository ID` | Permanent unique ID assigned to the repository by Praefect. Different to the ID GitLab uses for repositories. |
| `Virtual Storage` | Name of the virtual storage the repository is stored in. |
| `Relative Path` | Repository's path in the virtual storage. |
| `Replica Path` | Where on the Gitaly node's disk the repository's replicas are stored. |
| `Primary` | Current primary of the repository. |
| `Generation` | Used by Praefect to track repository changes. Each write in the repository increments the repository's generation. |
| `Replicas` | A list of replicas that exist or are expected to exist. |
For each replica, the following metadata is available:
| `Replicas` Field | Description |
|:-----------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `Storage` | Name of the Gitaly storage that contains the replica. |
| `Assigned` | Indicates whether the replica is expected to exist in the storage. Can be `false` if a Gitaly node is removed from the cluster or if the storage contains an extra copy after the repository's replication factor was decreased. |
| `Generation` | Latest confirmed generation of the replica. It indicates:<br><br>- The replica is fully up to date if the generation matches the repository's generation.<br>- The replica is outdated if the replica's generation is less than the repository's generation.<br>- `replica not yet created` if the replica does not yet exist at all on the storage. |
| `Healthy` | Indicates whether the Gitaly node that is hosting this replica is considered healthy by the consensus of Praefect nodes. |
| `Valid Primary` | Indicates whether the replica is fit to serve as the primary node. If the repository's primary is not a valid primary, a failover occurs on the next write to the repository if there is another replica that is a valid primary. A replica is a valid primary if:<br><br>- It is stored on a healthy Gitaly node.<br>- It is fully up to date.<br>- It is not targeted by a pending deletion job from decreasing replication factor.<br>- It is assigned. |
## Unavailable repositories
> - From GitLab 13.0 through 14.0, repositories became read-only if they were outdated on the primary but fully up to date on a healthy secondary. `dataloss` sub-command displays read-only repositories by default through these versions.

24
doc/administration/gitaly/troubleshooting.md
View file

@ -374,17 +374,23 @@ Here are common errors and potential causes:
### Determine primary Gitaly node
To determine the current primary Gitaly node for a specific Praefect node:
To determine the primary node of a repository:
- Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the
[`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json).
This is recommended.
- If you do not have Grafana set up, use the following command on each host of each
Praefect node:
- In GitLab 14.6 and later, use the [`praefect metadata`](praefect.md#retrieve-repository-metadata) subcommand.
- In GitLab 13.12 to GitLab 14.5 with [repository-specific primaries](praefect.md#repository-specific-primary-nodes),
use the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums).
- With legacy election strategies in GitLab 13.12 and earlier, the primary was the same for all repositories in a virtual storage.
To determine the current primary Gitaly node for a specific virtual storage:
```shell
curl localhost:9652/metrics | grep gitaly_praefect_primaries`
```
- Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the
[`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json).
This is recommended.
- If you do not have Grafana set up, use the following command on each host of each
Praefect node:
```shell
curl localhost:9652/metrics | grep gitaly_praefect_primaries`
```
### Check that repositories are in sync

63
doc/ci/pipelines/job_artifacts.md
View file

@ -48,7 +48,45 @@ is used.
If you run two types of pipelines (like branch and scheduled) for the same ref,
the pipeline that finishes later creates the job artifact.
For more examples, view the [keyword reference for the `.gitlab-ci.yml` file](../yaml/index.md#artifacts).
To disable artifact passing, define the job with empty [dependencies](../yaml/index.md#dependencies):
```yaml
job:
stage: build
script: make build
dependencies: []
```
You may want to create artifacts only for tagged releases to avoid filling the
build server storage with temporary build artifacts. For example, use [`rules`](../yaml/index.md#rules)
to create artifacts only for tags:
```yaml
default-job:
script:
- mvn test -U
rules:
- if: $CI_COMMIT_BRANCH
release-job:
script:
- mvn package -U
artifacts:
paths:
- target/*.war
rules:
- if: $CI_COMMIT_TAG
```
You can use wildcards for directories too. For example, if you want to get all the
files inside the directories that end with `xyz`:
```yaml
job:
artifacts:
paths:
- path/*xyz/*
```
### Use CI/CD variables to define the artifacts name
@ -153,6 +191,29 @@ artifacts:
- binaries/temp/**/*
```
### Add untracked files to artifacts
Use [`artifacts:untracked`](../yaml/index.md#artifactsuntracked) to add all Git untracked
files as artifacts (along with the paths defined in [`artifacts:paths`](../yaml/index.md#artifactspaths)).
Save all Git untracked files and files in `binaries`:
```yaml
artifacts:
untracked: true
paths:
- binaries/
```
Save all untracked files but [exclude](../yaml/index.md#artifactsexclude) `*.txt`:
```yaml
artifacts:
untracked: true
exclude:
- "*.txt"
```
## Download job artifacts
You can download job artifacts or view the job archive:

146
doc/ci/yaml/index.md
View file

@ -819,64 +819,42 @@ job:
#### `artifacts:paths`
Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly
link outside it. You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming))
patterns and:
link outside it.
- In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620),
[`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match).
- In GitLab Runner 12.10 and earlier, [`filepath.Match`](https://pkg.go.dev/path/filepath#Match).
**Keyword type**: Job keyword. You can use it only as part of a job or in the
[`default:` section](#default).
To restrict which jobs a specific job fetches artifacts from, see [dependencies](#dependencies).
**Possible inputs**:
Send all files in `binaries` and `.config`:
- An array of file paths, relative to the project directory.
- You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming))
patterns and:
- In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620),
[`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match).
- In GitLab Runner 12.10 and earlier, [`filepath.Match`](https://pkg.go.dev/path/filepath#Match).
```yaml
artifacts:
paths:
- binaries/
- .config
```
To disable artifact passing, define the job with empty [dependencies](#dependencies):
```yaml
job:
stage: build
script: make build
dependencies: []
```
You may want to create artifacts only for tagged releases to avoid filling the
build server storage with temporary build artifacts.
Create artifacts only for tags (`default-job` doesn't create artifacts):
```yaml
default-job:
script:
- mvn test -U
rules:
- if: $CI_COMMIT_BRANCH
release-job:
script:
- mvn package -U
artifacts:
paths:
- target/*.war
rules:
- if: $CI_COMMIT_TAG
```
You can use wildcards for directories too. For example, if you want to get all the files inside the directories that end with `xyz`:
**Example of `artifacts:paths`**:
```yaml
job:
artifacts:
paths:
- path/*xyz/*
- binaries/
- .config
```
This example creates an artifact with `.config` and all the files in the `binaries` directory.
**Additional details**:
- If not used with [`artifacts:name`](#artifactsname) defined, the artifacts file
is named `artifacts`, which becomes `artifacts.zip` when downloaded.
**Related topics**:
- To restrict which jobs a specific job fetches artifacts from, see [`dependencies`](#dependencies).
- [Create job artifacts](../pipelines/job_artifacts.md#create-job-artifacts).
#### `artifacts:public`
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8
@ -887,20 +865,25 @@ job:
Use `artifacts:public` to determine whether the job artifacts should be
publicly available.
The default for `artifacts:public` is `true` which means that the artifacts in
public pipelines are available for download by anonymous and guest users:
```yaml
artifacts:
public: true
```
When `artifacts:public` is `true` (default), the artifacts in
public pipelines are available for download by anonymous and guest users.
To deny read access for anonymous and guest users to artifacts in public
pipelines, set `artifacts:public` to `false`:
**Keyword type**: Job keyword. You can use it only as part of a job or in the
[`default:` section](#default).
**Possible inputs**:
- `true` (default if not defined) or `false`.
**Example of `artifacts:paths`**:
```yaml
artifacts:
public: false
job:
artifacts:
public: false
```
#### `artifacts:reports`
@ -941,45 +924,44 @@ Use `artifacts:untracked` to add all Git untracked files as artifacts (along
with the paths defined in `artifacts:paths`). `artifacts:untracked` ignores configuration
in the repository's `.gitignore` file.
Send all Git untracked files:
**Keyword type**: Job keyword. You can use it only as part of a job or in the
[`default:` section](#default).
**Possible inputs**:
- `true` or `false` (default if not defined).
**Example of `artifacts:untracked`**:
Save all Git untracked files:
```yaml
artifacts:
untracked: true
job:
artifacts:
untracked: true
```
Send all Git untracked files and files in `binaries`:
**Related topics**:
```yaml
artifacts:
untracked: true
paths:
- binaries/
```
Send all untracked files but [exclude](#artifactsexclude) `*.txt`:
```yaml
artifacts:
untracked: true
exclude:
- "*.txt"
```
- [Add untracked files to artifacts](../pipelines/job_artifacts.md#add-untracked-files-to-artifacts).
#### `artifacts:when`
Use `artifacts:when` to upload artifacts on job failure or despite the
failure.
`artifacts:when` can be set to one of the following values:
**Keyword type**: Job keyword. You can use it only as part of a job or in the
[`default:` section](#default).
1. `on_success` (default): Upload artifacts only when the job succeeds.
1. `on_failure`: Upload artifacts only when the job fails.
1. `always`: Always upload artifacts. For example, when
[uploading artifacts](../unit_test_reports.md#viewing-junit-screenshots-on-gitlab) required to
troubleshoot failing tests.
**Possible inputs**:
For example, to upload artifacts only when a job fails:
- `on_success` (default): Upload artifacts only when the job succeeds.
- `on_failure`: Upload artifacts only when the job fails.
- `always`: Always upload artifacts. For example, when
[uploading artifacts](../unit_test_reports.md#viewing-junit-screenshots-on-gitlab)
required to troubleshoot failing tests.
**Example of `artifacts:when`**:
```yaml
job:

17
doc/user/gitlab_com/index.md
View file

@ -143,6 +143,23 @@ the related documentation.
| [Max registered runners](../../administration/instance_limits.md#number-of-registered-runners-per-scope) | Free tier: `50` per-group / `50` per-project <br/> All paid tiers: `1_000` per-group / `1_000` per-project | `1_000` per-group / `1_000` per-project |
| [Limit dotenv variables](../../administration/instance_limits.md#limit-dotenv-variables) | Free tier: `50` / Premium tier: `100` / Ultimate tier: `150` | 150 |
## Package registry limits
The [maximum file size](../../administration/instance_limits.md#file-size-limits)
for a package uploaded to the [GitLab Package Registry](../../user/packages/package_registry/index.md)
varies by format:
| Package type | GitLab.com | Default |
|--------------|------------|---------|
| Conan | 5 GB | 3 GB |
| Generic | 5 GB | 5 GB |
| Helm | 5 MB | 5 MB |
| Maven | 5 GB | 3 GB |
| npm: | 5 GB | 500 MB |
| NuGet | 5 GB | 500 MB |
| PyPI | 5 GB | 3 GB |
| Terraform | 1 GB | 1 GB |
## Account and limit settings
GitLab.com has the following [account limits](../admin_area/settings/account_and_limit_settings.md)