2020-10-30 17:08:52 -04:00
---
stage: none
group: unassigned
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---
2018-09-04 13:20:30 -04:00
# End-to-end Testing
2017-12-20 08:05:02 -05:00
2018-09-04 13:20:30 -04:00
## What is end-to-end testing?
2017-12-20 08:05:02 -05:00
2018-09-04 13:20:30 -04:00
End-to-end testing is a strategy used to check whether your application works
as expected across the entire software stack and architecture, including
integration of all micro-services and components that are supposed to work
2017-12-20 08:05:02 -05:00
together.
## How do we test GitLab?
2020-04-14 08:09:26 -04:00
We use [Omnibus GitLab ](https://gitlab.com/gitlab-org/omnibus-gitlab ) to build GitLab packages and then we
test these packages using the [GitLab QA orchestrator ](https://gitlab.com/gitlab-org/gitlab-qa ) tool, which is
2018-09-04 13:20:30 -04:00
a black-box testing framework for the API and the UI.
2017-12-20 08:05:02 -05:00
### Testing nightly builds
2020-01-06 01:08:07 -05:00
We run scheduled pipelines each night to test nightly builds created by Omnibus.
2019-09-27 08:06:07 -04:00
You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/nightly/pipelines`
(need Developer access permissions). Results are reported in the `#qa-nightly` Slack channel.
2018-09-04 13:20:30 -04:00
### Testing staging
2020-01-06 01:08:07 -05:00
We run scheduled pipelines each night to test staging.
2019-09-27 08:06:07 -04:00
You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/staging/pipelines`
2020-01-06 01:08:07 -05:00
(need Developer access permissions). Results are reported in the `#qa-staging` Slack channel.
2017-12-20 08:05:02 -05:00
### Testing code in merge requests
2020-01-06 07:07:56 -05:00
#### Using the `package-and-qa` job
2019-03-25 07:27:03 -04:00
2019-03-15 01:10:56 -04:00
It is possible to run end-to-end tests for a merge request, eventually being run in
2020-04-06 02:09:19 -04:00
a pipeline in the [`gitlab-qa-mirror` ](https://gitlab.com/gitlab-org/gitlab-qa-mirror/ ) project,
2020-01-06 07:07:56 -05:00
by triggering the `package-and-qa` manual action in the `test` stage (not
2019-03-25 07:27:03 -04:00
available for forks).
2020-01-13 16:07:39 -05:00
**This runs end-to-end tests against a custom CE and EE (with an Ultimate license)
Omnibus package built from your merge request's changes.**
2018-01-02 08:22:48 -05:00
2018-01-11 07:05:55 -05:00
Manual action that starts end-to-end tests is also available in merge requests
2020-04-14 08:09:26 -04:00
in [Omnibus GitLab ](https://gitlab.com/gitlab-org/omnibus-gitlab ).
2017-12-20 08:05:02 -05:00
Below you can read more about how to use it and how does it work.
2018-01-02 08:22:48 -05:00
#### How does it work?
2018-01-04 06:09:14 -05:00
Currently, we are using _multi-project pipeline_ -like approach to run QA
pipelines.
2018-01-02 08:22:48 -05:00
2019-07-31 02:22:22 -04:00
```mermaid
2019-03-11 12:50:03 -04:00
graph LR
2020-04-06 02:09:19 -04:00
A1 -.->|1. Triggers an omnibus-gitlab-mirror pipeline and wait for it to be done| A2
B2[`Trigger-qa` stage< br > `Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa-mirror pipeline and wait for it to be done| A3
2019-03-11 12:50:03 -04:00
2019-09-18 14:06:14 -04:00
subgraph "gitlab-foss/gitlab pipeline"
2020-01-06 07:07:56 -05:00
A1[`test` stage< br > `package-and-qa` job]
2019-03-11 12:50:03 -04:00
end
2019-07-31 02:22:22 -04:00
subgraph "omnibus-gitlab pipeline"
A2[`Trigger-docker` stage< br > `Trigger:gitlab-docker` job] -->|once done| B2
2019-03-11 12:50:03 -04:00
end
2020-04-06 02:09:19 -04:00
subgraph "gitlab-qa-mirror pipeline"
2020-01-06 07:07:56 -05:00
A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa` job< br > and post the result on the original commit tested| A1
2019-03-11 12:50:03 -04:00
end
2019-07-31 02:22:22 -04:00
```
2020-05-25 11:07:58 -04:00
1. Developer triggers a manual action, that can be found in GitLab merge
2019-02-22 08:17:10 -05:00
requests. This starts a chain of pipelines in multiple projects.
2018-01-02 08:37:01 -05:00
2020-04-14 08:09:26 -04:00
1. The script being executed triggers a pipeline in
2020-05-25 11:07:58 -04:00
[Omnibus GitLab Mirror ](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror )
2019-02-22 08:17:10 -05:00
and waits for the resulting status. We call this a _status attribution_ .
2018-01-02 08:37:01 -05:00
2020-05-25 11:07:58 -04:00
1. GitLab packages are being built in the [Omnibus GitLab Mirror ](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror )
2019-02-22 08:17:10 -05:00
pipeline. Packages are then pushed to its Container Registry.
2018-01-02 08:37:01 -05:00
2018-01-02 08:22:48 -05:00
1. When packages are ready, and available in the registry, a final step in the
2020-05-25 11:07:58 -04:00
[Omnibus GitLab Mirror ](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror ) pipeline, triggers a new
2020-04-06 02:09:19 -04:00
GitLab QA pipeline (those with access can view them at `https://gitlab.com/gitlab-org/gitlab-qa-mirror/pipelines` ). It also waits for a resulting status.
2018-01-02 08:37:01 -05:00
2018-01-02 08:22:48 -05:00
1. GitLab QA pulls images from the registry, spins-up containers and runs tests
2019-02-22 08:17:10 -05:00
against a test environment that has been just orchestrated by the `gitlab-qa`
tool.
2018-01-02 08:37:01 -05:00
2019-04-29 05:33:08 -04:00
1. The result of the GitLab QA pipeline is being
2020-05-25 11:07:58 -04:00
propagated upstream, through Omnibus, back to the GitLab merge request.
2017-12-20 08:05:02 -05:00
2020-05-29 11:08:14 -04:00
Please note, we plan to [add more specific information ](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/156 )
2020-04-06 02:09:19 -04:00
about the tests included in each job/scenario that runs in `gitlab-qa-mirror` .
2020-01-20 04:08:32 -05:00
2020-02-28 01:09:19 -05:00
#### With Pipeline for Merged Results
In a Pipeline for Merged Results, the pipeline runs on a new ref that contains the merge result of the source and target branch.
2020-04-06 02:09:19 -04:00
However, this ref is not available to the `gitlab-qa-mirror` pipeline.
2020-02-28 01:09:19 -05:00
For this reason, the end-to-end tests on a Pipeline for Merged Results would use the head of the merge request source branch.
```mermaid
graph LR
A["a1b1c1 - branch HEAD (CI_MERGE_REQUEST_SOURCE_BRANCH_SHA)"]
B["x1y1z1 - master HEAD"]
C["d1e1f1 - merged results (CI_COMMIT_SHA)"]
A --> C
B --> C
A --> E["E2E tests"]
C --> D["Pipeline for merged results"]
```
2020-01-20 04:08:32 -05:00
##### Running custom tests
The [existing scenarios ](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md )
2020-04-06 02:09:19 -04:00
that run in the downstream `gitlab-qa-mirror` pipeline include many tests, but there are times when you might want to run a
2020-01-20 04:08:32 -05:00
test or a group of tests that are different than the groups in any of the existing scenarios.
For example, when we [dequarantine ](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#dequarantining-tests )
a flaky test we first want to make sure that it's no longer flaky.
We can do that using the `ce:custom-parallel` and `ee:custom-parallel` jobs.
Both are manual jobs that you can configure using custom variables.
2020-11-19 19:09:06 -05:00
When clicking the name (not the play icon) of one of the parallel jobs,
you are prompted to enter variables. You can use any of [the variables
2020-01-20 04:08:32 -05:00
that can be used with `gitlab-qa` ](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables)
as well as these:
| Variable | Description |
|-|-|
| `QA_SCENARIO` | The scenario to run (default `Test::Instance::Image` ) |
| `QA_TESTS` | The test(s) to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, e.g., `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. |
| `QA_RSPEC_TAGS` | The RSpec tags to add (no default) |
2020-11-19 19:09:06 -05:00
For now [manual jobs with custom variables don't use the same variable
2020-05-21 02:08:25 -04:00
when retried](https://gitlab.com/gitlab-org/gitlab/-/issues/31367), so if you want to run the same test(s) multiple times,
2020-01-20 04:08:32 -05:00
specify the same variables in each `custom-parallel` job (up to as
many of the 10 available jobs that you want to run).
2019-03-25 07:27:03 -04:00
#### Using the `review-qa-all` jobs
On every pipeline during the `test` stage, the `review-qa-smoke` job is
automatically started: it runs the QA smoke suite against the
2020-04-14 08:09:26 -04:00
[Review App ](../review_apps.md ).
2019-03-25 07:27:03 -04:00
You can also manually start the `review-qa-all` : it runs the full QA suite
2020-04-14 08:09:26 -04:00
against the [Review App ](../review_apps.md ).
2019-03-25 07:27:03 -04:00
**This runs end-to-end tests against a Review App based on [the official GitLab
2020-04-14 08:09:26 -04:00
Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/), itself deployed with custom
[Cloud Native components ](https://gitlab.com/gitlab-org/build/CNG ) built from your merge request's changes.**
2019-03-25 07:27:03 -04:00
2020-04-14 08:09:26 -04:00
See [Review Apps ](../review_apps.md ) for more details about Review Apps.
2019-03-25 07:27:03 -04:00
2019-06-18 20:22:23 -04:00
## How do I run the tests?
2020-01-20 04:08:32 -05:00
If you are not [testing code in a merge request ](#testing-code-in-merge-requests ),
there are two main options for running the tests. If you simply want to run
2020-05-27 05:08:30 -04:00
the existing tests against a live GitLab instance or against a pre-built Docker image
2020-04-14 08:09:26 -04:00
you can use the [GitLab QA orchestrator ](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md ). See also [examples
2019-06-18 20:22:23 -04:00
of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples).
On the other hand, if you would like to run against a local development GitLab
environment, you can use the [GitLab Development Kit (GDK) ](https://gitlab.com/gitlab-org/gitlab-development-kit/ ).
2019-09-18 10:02:45 -04:00
Please refer to the instructions in the [QA README ](https://gitlab.com/gitlab-org/gitlab/tree/master/qa/README.md#how-can-i-use-it )
2019-06-18 20:22:23 -04:00
and the section below.
2020-03-16 17:09:21 -04:00
### Running tests that require special setup
Learn how to perform [tests that require special setup or consideration to run on your local environment ](running_tests_that_require_special_setup.md ).
2019-03-25 07:27:03 -04:00
## How do I write tests?
2017-12-20 08:05:02 -05:00
2018-01-02 08:22:48 -05:00
In order to write new tests, you first need to learn more about GitLab QA
2020-04-14 08:09:26 -04:00
architecture. See the [documentation about it ](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md ).
2017-12-20 08:05:02 -05:00
2020-04-14 08:09:26 -04:00
Once you decided where to put [test environment orchestration scenarios ](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario ) and
[instance-level scenarios ](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features ), take a look at the [GitLab QA README ](https://gitlab.com/gitlab-org/gitlab/tree/master/qa/README.md ),
the [GitLab QA orchestrator README ](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md ), and [the already existing
instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features).
2017-12-20 08:05:02 -05:00
2020-08-17 17:09:56 -04:00
### Consider **not** writing an end-to-end test
We should follow these best practices for end-to-end tests:
- Do not write an end-to-end test if a lower-level feature test exists. End-to-end tests require more work and resources.
- Troubleshooting for end-to-end tests can be more complex as connections to the application under test are not known.
2019-07-12 04:15:38 -04:00
Continued reading:
2019-05-21 21:34:12 -04:00
2020-04-28 20:09:38 -04:00
- [Beginner's Guide ](beginners_guide.md )
2019-05-21 21:34:12 -04:00
- [Style Guide ](style_guide.md )
- [Best Practices ](best_practices.md )
2019-10-29 08:06:40 -04:00
- [Testing with feature flags ](feature_flags.md )
2019-11-12 01:06:32 -05:00
- [Flows ](flows.md )
2020-03-09 14:07:59 -04:00
- [RSpec metadata/tags ](rspec_metadata_tests.md )
2019-05-21 21:34:12 -04:00
2017-12-20 08:05:02 -05:00
## Where can I ask for help?
2018-04-19 03:28:08 -04:00
You can ask question in the `#quality` channel on Slack (GitLab internal) or
you can find an issue you would like to work on in
2020-05-29 11:08:14 -04:00
[the `gitlab` issue tracker ](https://gitlab.com/gitlab-org/gitlab/-/issues?label_name%5B%5D=QA&label_name%5B%5D=test ), or
[the `gitlab-qa` issue tracker ](https://gitlab.com/gitlab-org/gitlab-qa/-/issues?label_name%5B%5D=new+scenario ).