2018-09-04 17:20:30 +00:00
|
|
|
# End-to-end Testing
|
2017-12-20 13:05:02 +00:00
|
|
|
|
2018-09-04 17:20:30 +00:00
|
|
|
## What is end-to-end testing?
|
2017-12-20 13:05:02 +00:00
|
|
|
|
2018-09-04 17:20:30 +00:00
|
|
|
End-to-end testing is a strategy used to check whether your application works
|
|
|
|
as expected across the entire software stack and architecture, including
|
|
|
|
integration of all micro-services and components that are supposed to work
|
2017-12-20 13:05:02 +00:00
|
|
|
together.
|
|
|
|
|
|
|
|
## How do we test GitLab?
|
|
|
|
|
|
|
|
We use [Omnibus GitLab][omnibus-gitlab] to build GitLab packages and then we
|
2018-09-04 17:20:30 +00:00
|
|
|
test these packages using the [GitLab QA orchestrator][gitlab-qa] tool, which is
|
|
|
|
a black-box testing framework for the API and the UI.
|
2017-12-20 13:05:02 +00:00
|
|
|
|
|
|
|
### Testing nightly builds
|
|
|
|
|
|
|
|
We run scheduled pipeline each night to test nightly builds created by Omnibus.
|
2018-09-04 17:20:30 +00:00
|
|
|
You can find these nightly pipelines at [gitlab-org/quality/nightly/pipelines][quality-nightly-pipelines].
|
2019-03-11 16:50:03 +00:00
|
|
|
Results are reported in the `#qa-nightly` Slack channel.
|
2018-09-04 17:20:30 +00:00
|
|
|
|
|
|
|
### Testing staging
|
|
|
|
|
|
|
|
We run scheduled pipeline each night to test staging.
|
|
|
|
You can find these nightly pipelines at [gitlab-org/quality/staging/pipelines][quality-staging-pipelines].
|
2019-03-11 16:50:03 +00:00
|
|
|
Results are reported in the `#qa-staging` Slack channel.
|
2017-12-20 13:05:02 +00:00
|
|
|
|
|
|
|
### Testing code in merge requests
|
|
|
|
|
2019-03-15 05:10:56 +00:00
|
|
|
It is possible to run end-to-end tests for a merge request, eventually being run in
|
|
|
|
a pipeline in the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/) project,
|
|
|
|
by triggering the `package-and-qa` manual action in the `test` stage (which should
|
|
|
|
be present in a merge request widget, unless the merge request comes from a fork).
|
2018-01-02 13:22:48 +00:00
|
|
|
|
2018-01-11 12:05:55 +00:00
|
|
|
Manual action that starts end-to-end tests is also available in merge requests
|
2018-09-04 17:20:30 +00:00
|
|
|
in [Omnibus GitLab][omnibus-gitlab].
|
2017-12-20 13:05:02 +00:00
|
|
|
|
|
|
|
Below you can read more about how to use it and how does it work.
|
|
|
|
|
2018-01-02 13:22:48 +00:00
|
|
|
#### How does it work?
|
|
|
|
|
2018-01-04 11:09:14 +00:00
|
|
|
Currently, we are using _multi-project pipeline_-like approach to run QA
|
|
|
|
pipelines.
|
2018-01-02 13:22:48 +00:00
|
|
|
|
2019-03-11 16:50:03 +00:00
|
|
|
![QA on merge requests CI/CD architecture](img/qa_on_merge_requests_cicd_architecture.png)
|
|
|
|
|
|
|
|
<details>
|
|
|
|
<summary>Show mermaid source</summary>
|
|
|
|
<pre>
|
|
|
|
graph LR
|
|
|
|
A1 -.->|1. Triggers an omnibus-gitlab pipeline and wait for it to be done| A2
|
|
|
|
B2[<b>`Trigger-qa` stage</b><br />`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa pipeline and wait for it to be done| A3
|
|
|
|
|
|
|
|
subgraph gitlab-ce/ee pipeline
|
|
|
|
A1[<b>`test` stage</b><br />`package-and-qa` job]
|
|
|
|
end
|
|
|
|
|
|
|
|
subgraph omnibus-gitlab pipeline
|
|
|
|
A2[<b>`Trigger-docker` stage</b></b><br />`Trigger:gitlab-docker` job] -->|once done| B2
|
|
|
|
end
|
|
|
|
|
|
|
|
subgraph gitlab-qa pipeline
|
|
|
|
A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa` job<br />and post the result on the original commit tested| A1
|
|
|
|
end
|
|
|
|
</pre>
|
|
|
|
</details>
|
|
|
|
|
2018-09-04 17:20:30 +00:00
|
|
|
1. Developer triggers a manual action, that can be found in CE / EE merge
|
2019-02-22 13:17:10 +00:00
|
|
|
requests. This starts a chain of pipelines in multiple projects.
|
2018-01-02 13:37:01 +00:00
|
|
|
|
2018-09-04 17:20:30 +00:00
|
|
|
1. The script being executed triggers a pipeline in [Omnibus GitLab][omnibus-gitlab]
|
2019-02-22 13:17:10 +00:00
|
|
|
and waits for the resulting status. We call this a _status attribution_.
|
2018-01-02 13:37:01 +00:00
|
|
|
|
2018-09-04 17:20:30 +00:00
|
|
|
1. GitLab packages are being built in the [Omnibus GitLab][omnibus-gitlab]
|
2019-02-22 13:17:10 +00:00
|
|
|
pipeline. Packages are then pushed to its Container Registry.
|
2018-01-02 13:37:01 +00:00
|
|
|
|
2018-01-02 13:22:48 +00:00
|
|
|
1. When packages are ready, and available in the registry, a final step in the
|
2019-02-22 13:17:10 +00:00
|
|
|
[Omnibus GitLab][omnibus-gitlab] pipeline, triggers a new
|
|
|
|
[GitLab QA pipeline][gitlab-qa-pipelines]. It also waits for a resulting status.
|
2018-01-02 13:37:01 +00:00
|
|
|
|
2018-01-02 13:22:48 +00:00
|
|
|
1. GitLab QA pulls images from the registry, spins-up containers and runs tests
|
2019-02-22 13:17:10 +00:00
|
|
|
against a test environment that has been just orchestrated by the `gitlab-qa`
|
|
|
|
tool.
|
2018-01-02 13:37:01 +00:00
|
|
|
|
2018-09-04 17:20:30 +00:00
|
|
|
1. The result of the [GitLab QA pipeline][gitlab-qa-pipelines] is being
|
2019-02-22 13:17:10 +00:00
|
|
|
propagated upstream, through Omnibus, back to the CE / EE merge request.
|
2017-12-20 13:05:02 +00:00
|
|
|
|
2018-01-02 13:22:48 +00:00
|
|
|
#### How do I write tests?
|
2017-12-20 13:05:02 +00:00
|
|
|
|
2018-01-02 13:22:48 +00:00
|
|
|
In order to write new tests, you first need to learn more about GitLab QA
|
2018-09-04 17:20:30 +00:00
|
|
|
architecture. See the [documentation about it][gitlab-qa-architecture].
|
2017-12-20 13:05:02 +00:00
|
|
|
|
2018-09-04 17:20:30 +00:00
|
|
|
Once you decided where to put [test environment orchestration scenarios] and
|
|
|
|
[instance-level scenarios], take a look at the [GitLab QA README][instance-qa-readme],
|
|
|
|
the [GitLab QA orchestrator README][gitlab-qa-readme], and [the already existing
|
|
|
|
instance-level scenarios][instance-level scenarios].
|
2017-12-20 13:05:02 +00:00
|
|
|
|
|
|
|
## Where can I ask for help?
|
|
|
|
|
2018-04-19 07:28:08 +00:00
|
|
|
You can ask question in the `#quality` channel on Slack (GitLab internal) or
|
|
|
|
you can find an issue you would like to work on in
|
2018-09-04 17:20:30 +00:00
|
|
|
[the `gitlab-ce` issue tracker][gitlab-ce-issues],
|
|
|
|
[the `gitlab-ee` issue tracker][gitlab-ce-issues], or
|
|
|
|
[the `gitlab-qa` issue tracker][gitlab-qa-issues].
|
2017-12-20 13:05:02 +00:00
|
|
|
|
|
|
|
[omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab
|
|
|
|
[gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa
|
2018-09-04 17:20:30 +00:00
|
|
|
[gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md
|
|
|
|
[quality-nightly-pipelines]: https://gitlab.com/gitlab-org/quality/nightly/pipelines
|
|
|
|
[quality-staging-pipelines]: https://gitlab.com/gitlab-org/quality/staging/pipelines
|
2018-01-02 13:22:48 +00:00
|
|
|
[gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md
|
2018-09-04 17:20:30 +00:00
|
|
|
[gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues?label_name%5B%5D=new+scenario
|
|
|
|
[gitlab-ce-issues]: https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name[]=QA&label_name[]=test
|
|
|
|
[gitlab-ee-issues]: https://gitlab.com/gitlab-org/gitlab-ee/issues?label_name[]=QA&label_name[]=test
|
|
|
|
[test environment orchestration scenarios]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario
|
|
|
|
[instance-level scenarios]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/specs/features
|
|
|
|
[Page objects documentation]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page/README.md
|
2018-01-02 13:37:01 +00:00
|
|
|
[instance-qa-readme]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/README.md
|
|
|
|
[instance-qa-examples]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa
|