kotovalexarian-likes-gitlab
/
gitlab-org--gitlab-foss
1
0
Fork 0
Code Releases Activity
gitlab-org--gitlab-foss/doc/development/testing_guide/end_to_end/index.md

155 lines
6.8 KiB
Markdown
Raw Normal View History

Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
# End-to-end Testing
Add docs about end-to-end testing / GitLab QA tests
2017-12-20 13:05:02 +00:00
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
## What is end-to-end testing?
Add docs about end-to-end testing / GitLab QA tests
2017-12-20 13:05:02 +00:00
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
End-to-end testing is a strategy used to check whether your application works
as expected across the entire software stack and architecture, including
integration of all micro-services and components that are supposed to work
Add docs about end-to-end testing / GitLab QA tests
2017-12-20 13:05:02 +00:00
together.
## How do we test GitLab?
We use [Omnibus GitLab][omnibus-gitlab] to build GitLab packages and then we
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
test these packages using the [GitLab QA orchestrator][gitlab-qa] tool, which is
a black-box testing framework for the API and the UI.
Add docs about end-to-end testing / GitLab QA tests
2017-12-20 13:05:02 +00:00
### Testing nightly builds
Add latest changes from gitlab-org/gitlab@master
2020-01-06 06:08:07 +00:00
We run scheduled pipelines each night to test nightly builds created by Omnibus.
Add latest changes from gitlab-org/gitlab@master
2019-09-27 12:06:07 +00:00
You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/nightly/pipelines`
(need Developer access permissions). Results are reported in the `#qa-nightly` Slack channel.
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
### Testing staging
Add latest changes from gitlab-org/gitlab@master
2020-01-06 06:08:07 +00:00
We run scheduled pipelines each night to test staging.
Add latest changes from gitlab-org/gitlab@master
2019-09-27 12:06:07 +00:00
You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/staging/pipelines`
Add latest changes from gitlab-org/gitlab@master
2020-01-06 06:08:07 +00:00
(need Developer access permissions). Results are reported in the `#qa-staging` Slack channel.
Add docs about end-to-end testing / GitLab QA tests
2017-12-20 13:05:02 +00:00
### Testing code in merge requests
Add docs exceptions to two pipeline jobs
2019-08-22 11:56:14 +00:00
#### Using the `package-and-qa-manual` job
Mention review-qa-all in the end-to-end tests documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2019-03-25 11:27:03 +00:00
Docs: Fix a link that fails the link test
2019-03-15 05:10:56 +00:00
It is possible to run end-to-end tests for a merge request, eventually being run in
a pipeline in the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/) project,
Add docs exceptions to two pipeline jobs
2019-08-22 11:56:14 +00:00
by triggering the `package-and-qa-manual` manual action in the `test` stage (not
Mention review-qa-all in the end-to-end tests documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2019-03-25 11:27:03 +00:00
available for forks).
**This runs end-to-end tests against a custom Omnibus package built from your
merge request's changes.**
Extend documentation on end-to-end integration tests
2018-01-02 13:22:48 +00:00
Update end_to_end_tests.md
2018-01-11 12:05:55 +00:00
Manual action that starts end-to-end tests is also available in merge requests
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
in [Omnibus GitLab][omnibus-gitlab].
Add docs about end-to-end testing / GitLab QA tests
2017-12-20 13:05:02 +00:00
Below you can read more about how to use it and how does it work.
Extend documentation on end-to-end integration tests
2018-01-02 13:22:48 +00:00
#### How does it work?
Copy-edit end-to-end testing guide documentation
2018-01-04 11:09:14 +00:00
Currently, we are using _multi-project pipeline_-like approach to run QA
pipelines.
Extend documentation on end-to-end integration tests
2018-01-02 13:22:48 +00:00
Use Mermaid Markdown for testing guide docs * Remove PNGs in favor of dynamically rendered graphs * Eliminate <b> styling because this is no longer supported * Change <br /> to <br> because the latter is only supported * Add quotes around subgraph titles
2019-07-31 06:22:22 +00:00
```mermaid
Document better the package-and-qa job Signed-off-by: Rémy Coutable <remy@rymai.me>
2019-03-11 16:50:03 +00:00
graph LR
A1 -.->|1. Triggers an omnibus-gitlab pipeline and wait for it to be done| A2
Use Mermaid Markdown for testing guide docs * Remove PNGs in favor of dynamically rendered graphs * Eliminate <b> styling because this is no longer supported * Change <br /> to <br> because the latter is only supported * Add quotes around subgraph titles
2019-07-31 06:22:22 +00:00
B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa pipeline and wait for it to be done| A3
Document better the package-and-qa job Signed-off-by: Rémy Coutable <remy@rymai.me>
2019-03-11 16:50:03 +00:00
Add latest changes from gitlab-org/gitlab@master
2019-09-18 18:06:14 +00:00
subgraph "gitlab-foss/gitlab pipeline"
Add docs exceptions to two pipeline jobs
2019-08-22 11:56:14 +00:00
A1[`test` stage<br>`package-and-qa-manual` job]
Document better the package-and-qa job Signed-off-by: Rémy Coutable <remy@rymai.me>
2019-03-11 16:50:03 +00:00
end
Use Mermaid Markdown for testing guide docs * Remove PNGs in favor of dynamically rendered graphs * Eliminate <b> styling because this is no longer supported * Change <br /> to <br> because the latter is only supported * Add quotes around subgraph titles
2019-07-31 06:22:22 +00:00
subgraph "omnibus-gitlab pipeline"
A2[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2
Document better the package-and-qa job Signed-off-by: Rémy Coutable <remy@rymai.me>
2019-03-11 16:50:03 +00:00
end
Use Mermaid Markdown for testing guide docs * Remove PNGs in favor of dynamically rendered graphs * Eliminate <b> styling because this is no longer supported * Change <br /> to <br> because the latter is only supported * Add quotes around subgraph titles
2019-07-31 06:22:22 +00:00
subgraph "gitlab-qa pipeline"
Add docs exceptions to two pipeline jobs
2019-08-22 11:56:14 +00:00
A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa-manual` job<br>and post the result on the original commit tested| A1
Document better the package-and-qa job Signed-off-by: Rémy Coutable <remy@rymai.me>
2019-03-11 16:50:03 +00:00
end
Use Mermaid Markdown for testing guide docs * Remove PNGs in favor of dynamically rendered graphs * Eliminate <b> styling because this is no longer supported * Change <br /> to <br> because the latter is only supported * Add quotes around subgraph titles
2019-07-31 06:22:22 +00:00
```
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
1. Developer triggers a manual action, that can be found in CE / EE merge
Ensure all lists are surrounded by new lines Markdown renderers find it easier to determine where lists start and end when lists are surrounded by new lines. For consistency, also ensure entries in the list are aligned when they span multipls lines.
2019-02-22 13:17:10 +00:00
requests. This starts a chain of pipelines in multiple projects.
Copy-edit end-to-end testing guidelines
2018-01-02 13:37:01 +00:00
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
1. The script being executed triggers a pipeline in [Omnibus GitLab][omnibus-gitlab]
Ensure all lists are surrounded by new lines Markdown renderers find it easier to determine where lists start and end when lists are surrounded by new lines. For consistency, also ensure entries in the list are aligned when they span multipls lines.
2019-02-22 13:17:10 +00:00
and waits for the resulting status. We call this a _status attribution_.
Copy-edit end-to-end testing guidelines
2018-01-02 13:37:01 +00:00
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
1. GitLab packages are being built in the [Omnibus GitLab][omnibus-gitlab]
Ensure all lists are surrounded by new lines Markdown renderers find it easier to determine where lists start and end when lists are surrounded by new lines. For consistency, also ensure entries in the list are aligned when they span multipls lines.
2019-02-22 13:17:10 +00:00
pipeline. Packages are then pushed to its Container Registry.
Copy-edit end-to-end testing guidelines
2018-01-02 13:37:01 +00:00
Extend documentation on end-to-end integration tests
2018-01-02 13:22:48 +00:00
1. When packages are ready, and available in the registry, a final step in the
Ensure all lists are surrounded by new lines Markdown renderers find it easier to determine where lists start and end when lists are surrounded by new lines. For consistency, also ensure entries in the list are aligned when they span multipls lines.
2019-02-22 13:17:10 +00:00
[Omnibus GitLab][omnibus-gitlab] pipeline, triggers a new
Avoid links requiring permissions
2019-04-29 09:33:08 +00:00
GitLab QA pipeline (those with access can view them at `https://gitlab.com/gitlab-org/gitlab-qa/pipelines`). It also waits for a resulting status.
Copy-edit end-to-end testing guidelines
2018-01-02 13:37:01 +00:00
Extend documentation on end-to-end integration tests
2018-01-02 13:22:48 +00:00
1. GitLab QA pulls images from the registry, spins-up containers and runs tests
Ensure all lists are surrounded by new lines Markdown renderers find it easier to determine where lists start and end when lists are surrounded by new lines. For consistency, also ensure entries in the list are aligned when they span multipls lines.
2019-02-22 13:17:10 +00:00
against a test environment that has been just orchestrated by the `gitlab-qa`
tool.
Copy-edit end-to-end testing guidelines
2018-01-02 13:37:01 +00:00
Avoid links requiring permissions
2019-04-29 09:33:08 +00:00
1. The result of the GitLab QA pipeline is being
Ensure all lists are surrounded by new lines Markdown renderers find it easier to determine where lists start and end when lists are surrounded by new lines. For consistency, also ensure entries in the list are aligned when they span multipls lines.
2019-02-22 13:17:10 +00:00
propagated upstream, through Omnibus, back to the CE / EE merge request.
Add docs about end-to-end testing / GitLab QA tests
2017-12-20 13:05:02 +00:00
Mention review-qa-all in the end-to-end tests documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2019-03-25 11:27:03 +00:00
#### Using the `review-qa-all` jobs
On every pipeline during the `test` stage, the `review-qa-smoke` job is
automatically started: it runs the QA smoke suite against the
[Review App][review-apps].
You can also manually start the `review-qa-all`: it runs the full QA suite
against the [Review App][review-apps].
**This runs end-to-end tests against a Review App based on [the official GitLab
Helm chart][helm-chart], itself deployed with custom
[Cloud Native components][cng] built from your merge request's changes.**
See [Review Apps][review-apps] for more details about Review Apps.
Use new location for helm charts repo Signed-off-by: Balasankar "Balu" C <balasankar@gitlab.com>
2019-09-03 04:36:02 +00:00
[helm-chart]: https://gitlab.com/gitlab-org/charts/gitlab/
Mention review-qa-all in the end-to-end tests documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2019-03-25 11:27:03 +00:00
[cng]: https://gitlab.com/gitlab-org/build/CNG
Update docs on how to run E2E tests Make instructions on how to run the E2E tests against GDK more explicit and easier to find.
2019-06-19 00:22:23 +00:00
## How do I run the tests?
There are two main options for running the tests. If you simply want to run the
existing tests against a live GitLab instance or against a pre-built docker image
you can use the [GitLab QA orchestrator][gitlab-qa-readme]. See also [examples
of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples).
On the other hand, if you would like to run against a local development GitLab
environment, you can use the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/).
Add latest changes from gitlab-org/gitlab@master
2019-09-18 14:02:45 +00:00
Please refer to the instructions in the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa/README.md#how-can-i-use-it)
Update docs on how to run E2E tests Make instructions on how to run the E2E tests against GDK more explicit and easier to find.
2019-06-19 00:22:23 +00:00
and the section below.
Mention review-qa-all in the end-to-end tests documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2019-03-25 11:27:03 +00:00
## How do I write tests?
Add docs about end-to-end testing / GitLab QA tests
2017-12-20 13:05:02 +00:00
Extend documentation on end-to-end integration tests
2018-01-02 13:22:48 +00:00
In order to write new tests, you first need to learn more about GitLab QA
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
architecture. See the [documentation about it][gitlab-qa-architecture].
Add docs about end-to-end testing / GitLab QA tests
2017-12-20 13:05:02 +00:00
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
Once you decided where to put [test environment orchestration scenarios] and
[instance-level scenarios], take a look at the [GitLab QA README][instance-qa-readme],
the [GitLab QA orchestrator README][gitlab-qa-readme], and [the already existing
instance-level scenarios][instance-level scenarios].
Add docs about end-to-end testing / GitLab QA tests
2017-12-20 13:05:02 +00:00
Remove trailing whitespace in docs Remove unneeded trailing whitespace in lines in /development /ci /api and /administration docs.
2019-07-12 08:15:38 +00:00
Continued reading:
Consolidate documentation written for e2e tests under dev docs Previously the documentation was separated. If we want to bake quality into the product, how better than to include everything we use directly in the development documentation Signed-off-by: ddavison <ddavison@gitlab.com> Fix broken internal doc link Move documentation for page objects to the e2e section
2019-05-22 01:34:12 +00:00
- [Quick Start Guide](quick_start_guide.md)
- [Style Guide](style_guide.md)
- [Best Practices](best_practices.md)
Add latest changes from gitlab-org/gitlab@master
2019-10-29 12:06:40 +00:00
- [Testing with feature flags](feature_flags.md)
Add latest changes from gitlab-org/gitlab@master
2019-11-12 06:06:32 +00:00
- [Flows](flows.md)
Consolidate documentation written for e2e tests under dev docs Previously the documentation was separated. If we want to bake quality into the product, how better than to include everything we use directly in the development documentation Signed-off-by: ddavison <ddavison@gitlab.com> Fix broken internal doc link Move documentation for page objects to the e2e section
2019-05-22 01:34:12 +00:00
Add docs about end-to-end testing / GitLab QA tests
2017-12-20 13:05:02 +00:00
## Where can I ask for help?
Rename #qa slack channel to #quality in docs
2018-04-19 07:28:08 +00:00
You can ask question in the `#quality` channel on Slack (GitLab internal) or
you can find an issue you would like to work on in
Add latest changes from gitlab-org/gitlab@master
2019-09-18 18:06:14 +00:00
[the `gitlab` issue tracker][gitlab-issues], or
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
[the `gitlab-qa` issue tracker][gitlab-qa-issues].
Add docs about end-to-end testing / GitLab QA tests
2017-12-20 13:05:02 +00:00
[omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab
[gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
[gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md
Consolidate documentation written for e2e tests under dev docs Previously the documentation was separated. If we want to bake quality into the product, how better than to include everything we use directly in the development documentation Signed-off-by: ddavison <ddavison@gitlab.com> Fix broken internal doc link Move documentation for page objects to the e2e section
2019-05-22 01:34:12 +00:00
[review-apps]: ../review_apps.md
Extend documentation on end-to-end integration tests
2018-01-02 13:22:48 +00:00
[gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
[gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues?label_name%5B%5D=new+scenario
Add latest changes from gitlab-org/gitlab@master
2019-09-18 18:06:14 +00:00
[gitlab-issues]: https://gitlab.com/gitlab-org/gitlab/issues?label_name[]=QA&label_name[]=test
Improve the 'Testing levels' documentation Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-09-04 17:20:30 +00:00
[test environment orchestration scenarios]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario
Add latest changes from gitlab-org/gitlab@master
2019-09-18 14:02:45 +00:00
[instance-level scenarios]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features
[Page objects documentation]: https://gitlab.com/gitlab-org/gitlab/tree/master/qa/qa/page/README.md
[instance-qa-readme]: https://gitlab.com/gitlab-org/gitlab/tree/master/qa/README.md
[instance-qa-examples]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa