2020-10-30 21:08:52 +00:00
---
stage: none
group: unassigned
2022-09-21 21:13:33 +00:00
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
2020-10-30 21:08:52 +00:00
---
2018-09-04 17:20:30 +00:00
# End-to-end Testing
2017-12-20 13:05:02 +00:00
2018-09-04 17:20:30 +00:00
## What is end-to-end testing?
2017-12-20 13:05:02 +00:00
2022-08-30 12:12:25 +00:00
End-to-end (e2e) testing is a strategy used to check whether your application works
2018-09-04 17:20:30 +00:00
as expected across the entire software stack and architecture, including
integration of all micro-services and components that are supposed to work
2017-12-20 13:05:02 +00:00
together.
## How do we test GitLab?
2022-09-15 18:10:36 +00:00
We use [Omnibus GitLab ](https://gitlab.com/gitlab-org/omnibus-gitlab ) to build GitLab packages and then we test these packages
2022-08-30 12:12:25 +00:00
using the [GitLab QA orchestrator ](https://gitlab.com/gitlab-org/gitlab-qa ) tool to run the end-to-end tests located in the `qa` directory.
2017-12-20 13:05:02 +00:00
### Testing nightly builds
2020-01-06 06:08:07 +00:00
We run scheduled pipelines each night to test nightly builds created by Omnibus.
2021-03-30 18:10:47 +00:00
You can find these pipelines at < https: / / gitlab . com / gitlab-org / quality / nightly / pipelines >
2021-07-08 18:09:32 +00:00
(requires the Developer role). Results are reported in the `#qa-nightly` Slack channel.
2018-09-04 17:20:30 +00:00
### Testing staging
2020-01-06 06:08:07 +00:00
We run scheduled pipelines each night to test staging.
2021-03-30 18:10:47 +00:00
You can find these pipelines at < https: / / gitlab . com / gitlab-org / quality / staging / pipelines >
2021-07-08 18:09:32 +00:00
(requires the Developer role). Results are reported in the `#qa-staging` Slack channel.
2017-12-20 13:05:02 +00:00
### Testing code in merge requests
2022-08-30 12:12:25 +00:00
#### Using the package-and-test job
2019-03-25 11:27:03 +00:00
2022-08-30 12:12:25 +00:00
It is possible to run end-to-end tests for a merge request by triggering the `e2e:package-and-test` manual action in the `qa` stage (not
2019-03-25 11:27:03 +00:00
available for forks).
2021-03-30 18:10:47 +00:00
**This runs end-to-end tests against a custom EE (with an Ultimate license)
Docker image built from your merge request's changes.**
2018-01-02 13:22:48 +00:00
2021-03-30 18:10:47 +00:00
Manual action that starts end-to-end tests is also available
in [`gitlab-org/omnibus-gitlab` merge requests ](https://docs.gitlab.com/omnibus/build/team_member_docs.html#i-have-an-mr-in-the-omnibus-gitlab-project-and-want-a-package-or-docker-image-to-test-it ).
2017-12-20 13:05:02 +00:00
2018-01-02 13:22:48 +00:00
#### How does it work?
2021-03-30 18:10:47 +00:00
Currently, we are using _multi-project pipeline_ -like approach to run end-to-end
2018-01-04 11:09:14 +00:00
pipelines.
2018-01-02 13:22:48 +00:00
2019-07-31 06:22:22 +00:00
```mermaid
2021-03-30 18:10:47 +00:00
graph TB
A1 -.->|once done, can be triggered| A2
A2 -.->|1. Triggers an `omnibus-gitlab-mirror` pipeline< br > and wait for it to be done| B1
B2[`Trigger-qa` stage< br > `Trigger:qa-test` job] -.->|2. Triggers a `gitlab-qa-mirror` pipeline< br > and wait for it to be done| C1
subgraph "`gitlab-org/gitlab` pipeline"
A1[`build-images` stage< br > `build-qa-image` and `build-assets-image` jobs]
2022-08-31 09:13:12 +00:00
A2[`qa` stage< br > `e2e:package-and-test` job]
2019-03-11 16:50:03 +00:00
end
2021-03-30 18:10:47 +00:00
subgraph "`gitlab-org/build/omnibus-gitlab-mirror` pipeline"
B1[`Trigger-docker` stage< br > `Trigger:gitlab-docker` job] -->|once done| B2
2019-03-11 16:50:03 +00:00
end
2021-03-30 18:10:47 +00:00
subgraph "`gitlab-org/gitlab-qa-mirror` pipeline"
C1>End-to-end jobs run]
2019-03-11 16:50:03 +00:00
end
2019-07-31 06:22:22 +00:00
```
2021-03-30 18:10:47 +00:00
1. In the [`gitlab-org/gitlab` pipeline ](https://gitlab.com/gitlab-org/gitlab ):
2022-08-30 12:12:25 +00:00
1. Developer triggers the `e2e:package-and-test` manual action (available once the `build-qa-image` and
2021-03-30 18:10:47 +00:00
`build-assets-image` jobs are done), that can be found in GitLab merge
2022-08-30 12:12:25 +00:00
requests. This starts a e2e test child pipeline.
1. E2E child pipeline triggers a downstream pipeline in
2021-03-30 18:10:47 +00:00
[`gitlab-org/build/omnibus-gitlab-mirror` ](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror )
and polls for the resulting status. We call this a _status attribution_ .
2018-01-02 13:37:01 +00:00
2021-03-30 18:10:47 +00:00
1. In the [`gitlab-org/build/omnibus-gitlab-mirror` pipeline ](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror ):
1. Docker image is being built and pushed to its Container Registry.
2022-08-30 12:12:25 +00:00
1. Once Docker images are built and pushed jobs in `test` stage are started
2018-01-02 13:37:01 +00:00
2022-08-30 12:12:25 +00:00
1. In the `Test` stage:
2021-03-30 18:10:47 +00:00
1. Container for the Docker image stored in the [`gitlab-org/build/omnibus-gitlab-mirror` ](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror ) registry is spun-up.
1. End-to-end tests are run with the `gitlab-qa` executable, which spin up a container for the end-to-end image from the [`gitlab-org/gitlab` ](https://gitlab.com/gitlab-org/gitlab ) registry.
2018-01-02 13:37:01 +00:00
2021-04-06 12:09:21 +00:00
NOTE:
You may have noticed that we use `gitlab-org/build/omnibus-gitlab-mirror` instead of
2022-08-30 12:12:25 +00:00
`gitlab-org/omnibus-gitlab` .
2021-04-06 12:09:21 +00:00
This is due to technical limitations in the GitLab permission model: the ability to run a pipeline
against a protected branch is controlled by the ability to push/merge to this branch.
This means that for developers to be able to trigger a pipeline for the default branch in
2022-08-30 12:12:25 +00:00
`gitlab-org/omnibus-gitlab` , they would need to have the Maintainer role for this project.
2021-04-06 12:09:21 +00:00
For security reasons and implications, we couldn't open up the default branch to all the Developers.
2022-08-30 12:12:25 +00:00
Hence we created this mirror where Developers and Maintainers are allowed to push/merge to the default branch.
2021-04-06 12:09:21 +00:00
This problem was discovered in < https: / / gitlab . com / gitlab-org / gitlab-qa / - / issues / 63 # note_107175160 > and the "mirror"
work-around was suggested in < https: / / gitlab . com / gitlab-org / omnibus-gitlab / - / issues / 4717 > .
A feature proposal to segregate access control regarding running pipelines from ability to push/merge was also created at < https: / / gitlab . com / gitlab-org / gitlab / - / issues / 24585 > .
2022-01-27 09:14:40 +00:00
#### With merged results pipelines
2020-02-28 06:09:19 +00:00
2022-01-27 09:14:40 +00:00
In a merged results pipeline, the pipeline runs on a new ref that contains the merge result of the source and target branch.
2020-02-28 06:09:19 +00:00
2022-09-02 09:10:27 +00:00
The end-to-end tests on a merged results pipeline would use the new ref instead of the head of the merge request source branch.
2020-02-28 06:09:19 +00:00
```mermaid
graph LR
2022-09-02 09:10:27 +00:00
A["x1y1z1 - master HEAD"]
B["d1e1f1 - merged results (CI_COMMIT_SHA)"]
2020-02-28 06:09:19 +00:00
2022-09-02 09:10:27 +00:00
A --> B
2020-02-28 06:09:19 +00:00
2022-09-02 09:10:27 +00:00
B --> C["Merged results pipeline"]
C --> D["E2E tests"]
2020-02-28 06:09:19 +00:00
```
2020-01-20 09:08:32 +00:00
##### Running custom tests
The [existing scenarios ](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md )
2020-04-06 06:09:19 +00:00
that run in the downstream `gitlab-qa-mirror` pipeline include many tests, but there are times when you might want to run a
2020-01-20 09:08:32 +00:00
test or a group of tests that are different than the groups in any of the existing scenarios.
2022-03-30 12:09:14 +00:00
For example, when we [dequarantine ](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/debugging-qa-test-failures/#dequarantining-tests )
2020-01-20 09:08:32 +00:00
a flaky test we first want to make sure that it's no longer flaky.
We can do that using the `ce:custom-parallel` and `ee:custom-parallel` jobs.
Both are manual jobs that you can configure using custom variables.
2022-10-12 18:10:44 +00:00
When selecting the name (not the play icon) of one of the parallel jobs,
2022-08-24 00:12:25 +00:00
you are prompted to enter variables. You can use any of
2022-08-08 03:09:21 +00:00
[the variables that can be used with `gitlab-qa` ](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables )
2020-01-20 09:08:32 +00:00
as well as these:
| Variable | Description |
|-|-|
| `QA_SCENARIO` | The scenario to run (default `Test::Instance::Image` ) |
2022-05-13 18:08:33 +00:00
| `QA_TESTS` | The tests to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, for example, `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. |
2020-01-20 09:08:32 +00:00
| `QA_RSPEC_TAGS` | The RSpec tags to add (no default) |
2022-08-24 00:12:25 +00:00
For now,
[manual jobs with custom variables don't use the same variable when retried ](https://gitlab.com/gitlab-org/gitlab/-/issues/31367 ),
2022-08-08 03:09:21 +00:00
so if you want to run the same tests multiple times,
2020-01-20 09:08:32 +00:00
specify the same variables in each `custom-parallel` job (up to as
many of the 10 available jobs that you want to run).
2019-03-25 11:27:03 +00:00
#### Using the `review-qa-all` jobs
On every pipeline during the `test` stage, the `review-qa-smoke` job is
automatically started: it runs the QA smoke suite against the
2020-04-14 12:09:26 +00:00
[Review App ](../review_apps.md ).
2019-03-25 11:27:03 +00:00
You can also manually start the `review-qa-all` : it runs the full QA suite
2020-04-14 12:09:26 +00:00
against the [Review App ](../review_apps.md ).
2019-03-25 11:27:03 +00:00
2022-08-24 00:12:25 +00:00
**This runs end-to-end tests against a Review App based on
2022-08-08 03:09:21 +00:00
[the official GitLab Helm chart ](https://gitlab.com/gitlab-org/charts/gitlab/ ), itself deployed with custom
2020-04-14 12:09:26 +00:00
[Cloud Native components ](https://gitlab.com/gitlab-org/build/CNG ) built from your merge request's changes.**
2019-03-25 11:27:03 +00:00
2020-04-14 12:09:26 +00:00
See [Review Apps ](../review_apps.md ) for more details about Review Apps.
2019-03-25 11:27:03 +00:00
2022-01-08 00:14:32 +00:00
### Run tests in parallel
To run tests in parallel on CI, the [Knapsack ](https://github.com/KnapsackPro/knapsack )
gem is used. Knapsack reports are generated automatically and stored in the `GCS` bucket
2022-07-05 12:09:46 +00:00
`knapsack-reports` in the `gitlab-qa-resources` project. The [`KnapsackReport` ](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/support/knapsack_report.rb )
2022-01-08 00:14:32 +00:00
helper handles automated report generation and upload.
2022-01-06 18:13:45 +00:00
## Test metrics
For additional test health visibility, use a custom setup to export test execution
results to your [InfluxDb ](https://influxdb.quality.gitlab.net/ ) instance, and visualize
results as [Grafana ](https://dashboards.quality.gitlab.net/ ) dashboards.
### Provisioning
Provisioning of all components is performed by the
[`engineering-productivity-infrastructure` ](https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure ) project.
### Exporting metrics in CI
Use these environment variables to configure metrics export:
| Variable | Required | Information |
| -------- | -------- | ----------- |
| `QA_INFLUXDB_URL` | `true` | Should be set to `https://influxdb.quality.gitlab.net` . No default value. |
| `QA_INFLUXDB_TOKEN` | `true` | InfluxDB write token that can be found under `Influxdb auth tokens` document in `Gitlab-QA` `1Password` vault. No default value. |
2022-08-31 09:13:12 +00:00
| `QA_RUN_TYPE` | `false` | Arbitrary name for test execution, like `package-and-test` . Automatically inferred from the project name for live environment test executions. No default value. |
2022-01-06 18:13:45 +00:00
| `QA_EXPORT_TEST_METRICS` | `false` | Flag to enable or disable metrics export. Defaults to `true` . |
2021-09-14 12:10:35 +00:00
## Test reports
### Allure report
For additional test results visibility, tests that run on pipelines generate
and host [Allure ](https://github.com/allure-framework/allure2 ) test reports.
The `QA` framework is using the [Allure RSpec ](https://github.com/allure-framework/allure-ruby/blob/master/allure-rspec/README.md )
gem to generate source files for the `Allure` test report. An additional job
in the pipeline:
- Fetches these source files from all test jobs.
- Generates and uploads the report to the `GCS` bucket `gitlab-qa-allure-report` under the project `gitlab-qa-resources` .
2021-10-13 18:12:40 +00:00
A common CI template for report uploading is stored in
2021-09-14 12:10:35 +00:00
[`allure-report.yml` ](https://gitlab.com/gitlab-org/quality/pipeline-common/-/blob/master/ci/allure-report.yml ).
#### Merge requests
When these tests are executed in the scope of merge requests, the `Allure` report is
uploaded to the `GCS` bucket and comment is added linking to their respective reports.
#### Scheduled pipelines
Scheduled pipelines for these tests contain a `generate-allure-report` job under the `Report` stage. They also output
a link to the current test report.
#### Static report links
Each type of scheduled pipeline generates a static link for the latest test report according to its stage:
2022-08-30 12:12:25 +00:00
- [`master` ](https://storage.googleapis.com/gitlab-qa-allure-reports/e2e-package-and-test/master/index.html )
2021-09-14 12:10:35 +00:00
- [`staging-full` ](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-full/master/index.html )
- [`staging-sanity` ](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity/master/index.html )
- [`staging-sanity-no-admin` ](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity-no-admin/master/index.html )
- [`canary-sanity` ](https://storage.googleapis.com/gitlab-qa-allure-reports/canary-sanity/master/index.html )
2022-08-04 15:11:28 +00:00
- [`production` ](https://storage.googleapis.com/gitlab-qa-allure-reports/production-full/master/index.html )
2021-09-14 12:10:35 +00:00
- [`production-sanity` ](https://storage.googleapis.com/gitlab-qa-allure-reports/production-sanity/master/index.html )
2019-06-19 00:22:23 +00:00
## How do I run the tests?
2020-01-20 09:08:32 +00:00
If you are not [testing code in a merge request ](#testing-code-in-merge-requests ),
2021-04-01 06:09:29 +00:00
there are two main options for running the tests. If you want to run
the existing tests against a live GitLab instance or against a pre-built Docker image,
2022-08-24 00:12:25 +00:00
use the [GitLab QA orchestrator ](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md ). See also
2022-08-08 03:09:21 +00:00
[examples of the test scenarios you can run via the orchestrator ](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples ).
2019-06-19 00:22:23 +00:00
On the other hand, if you would like to run against a local development GitLab
environment, you can use the [GitLab Development Kit (GDK) ](https://gitlab.com/gitlab-org/gitlab-development-kit/ ).
2021-06-08 18:10:23 +00:00
Please refer to the instructions in the [QA README ](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/README.md#how-can-i-use-it )
2019-06-19 00:22:23 +00:00
and the section below.
2020-03-16 21:09:21 +00:00
### Running tests that require special setup
Learn how to perform [tests that require special setup or consideration to run on your local environment ](running_tests_that_require_special_setup.md ).
2019-03-25 11:27:03 +00:00
## How do I write tests?
2017-12-20 13:05:02 +00:00
2018-01-02 13:22:48 +00:00
In order to write new tests, you first need to learn more about GitLab QA
2020-04-14 12:09:26 +00:00
architecture. See the [documentation about it ](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md ).
2017-12-20 13:05:02 +00:00
2020-04-14 12:09:26 +00:00
Once you decided where to put [test environment orchestration scenarios ](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario ) and
2021-06-08 18:10:23 +00:00
[instance-level scenarios ](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features ), take a look at the [GitLab QA README ](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/README.md ),
2022-08-24 00:12:25 +00:00
the [GitLab QA orchestrator README ](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md ),
2022-08-08 03:09:21 +00:00
and [the already existing instance-level scenarios ](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features ).
2017-12-20 13:05:02 +00:00
2020-08-17 21:09:56 +00:00
### Consider **not** writing an end-to-end test
We should follow these best practices for end-to-end tests:
- Do not write an end-to-end test if a lower-level feature test exists. End-to-end tests require more work and resources.
- Troubleshooting for end-to-end tests can be more complex as connections to the application under test are not known.
2019-07-12 08:15:38 +00:00
Continued reading:
2019-05-22 01:34:12 +00:00
2020-04-29 00:09:38 +00:00
- [Beginner's Guide ](beginners_guide.md )
2019-05-22 01:34:12 +00:00
- [Style Guide ](style_guide.md )
- [Best Practices ](best_practices.md )
2019-10-29 12:06:40 +00:00
- [Testing with feature flags ](feature_flags.md )
2019-11-12 06:06:32 +00:00
- [Flows ](flows.md )
2020-03-09 18:07:59 +00:00
- [RSpec metadata/tags ](rspec_metadata_tests.md )
2021-07-19 15:09:40 +00:00
- [Execution context selection ](execution_context_selection.md )
2022-04-12 21:10:16 +00:00
- [Troubleshooting ](troubleshooting.md )
2019-05-22 01:34:12 +00:00
2017-12-20 13:05:02 +00:00
## Where can I ask for help?
2018-04-19 07:28:08 +00:00
You can ask question in the `#quality` channel on Slack (GitLab internal) or
you can find an issue you would like to work on in
2020-05-29 15:08:14 +00:00
[the `gitlab` issue tracker ](https://gitlab.com/gitlab-org/gitlab/-/issues?label_name%5B%5D=QA&label_name%5B%5D=test ), or
[the `gitlab-qa` issue tracker ](https://gitlab.com/gitlab-org/gitlab-qa/-/issues?label_name%5B%5D=new+scenario ).