From 1f50eb4f57ae14336a677f53d9a2c9501033f966 Mon Sep 17 00:00:00 2001 From: Grzegorz Bizon Date: Wed, 20 Dec 2017 14:05:02 +0100 Subject: [PATCH 1/7] Add docs about end-to-end testing / GitLab QA tests --- .../testing_guide/end_to_end_tests.md | 42 +++++++++++++++++++ doc/development/testing_guide/index.md | 8 ++++ 2 files changed, 50 insertions(+) create mode 100644 doc/development/testing_guide/end_to_end_tests.md diff --git a/doc/development/testing_guide/end_to_end_tests.md b/doc/development/testing_guide/end_to_end_tests.md new file mode 100644 index 00000000000..87a21b9f7fc --- /dev/null +++ b/doc/development/testing_guide/end_to_end_tests.md @@ -0,0 +1,42 @@ +# End-to-End Testing + +## What is End-to-End testing? + +End-to-End testing is a strategy used to check whether your application works +as expected across entire software stack and architecture, including +integration of all microservices and components that are supposed to work +together. + +## How do we test GitLab? + +We use [Omnibus GitLab][omnibus-gitlab] to build GitLab packages and then we +test these packages using [GitLab QA][gitlab-qa] project, which is entirely +black-box, click-driven testing framework. + +### Testing nightly builds + +We run scheduled pipeline each night to test nightly builds created by Omnibus. +You can find these nightly pipelines at [GitLab QA pipelines page][gitlab-qa-pipelines]. + +### Testing code in merge requests + +It is also possible to trigger packages build and [GitLab QA pipeline][gitlab-qa-pipelines] +using a manual action that should be present in the merge request widget on +your merge request. Look for `package-qa` manual action. + +Below you can read more about how to use it and how does it work. + +## How does it work? + +We are using _multi-project pipelines_ to run end-to-end tests. + +## How do I test my code? + +## How do I contribute? + +## Where can I ask for help? + + +[omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab +[gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa +[gitlab-qa-pipelines]: https://gitlab.com/gitlab-org/gitlab-qa/pipelines diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 65386f231a0..4ca192aee7e 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -65,6 +65,13 @@ Everything you should know about how to test Rake tasks. --- +## [End-to-end tests](end_to_end_tests.md) + +Everything you should know about how to run end-to-end tests, also known as +[GitLab QA][gitlab-qa] tests. + +--- + ## Spinach (feature) tests GitLab [moved from Cucumber to Spinach](https://github.com/gitlabhq/gitlabhq/pull/1426) @@ -89,3 +96,4 @@ test should be re-implemented using RSpec instead. [Capybara]: https://github.com/teamcapybara/capybara [Karma]: http://karma-runner.github.io/ [Jasmine]: https://jasmine.github.io/ +[gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa From 629d966c4807174ddb91e753b9aa15ec2697cd08 Mon Sep 17 00:00:00 2001 From: Grzegorz Bizon Date: Tue, 2 Jan 2018 14:22:48 +0100 Subject: [PATCH 2/7] Extend documentation on end-to-end integration tests --- .../testing_guide/end_to_end_tests.md | 47 ++++++++++++++++--- doc/development/testing_guide/index.md | 4 +- 2 files changed, 42 insertions(+), 9 deletions(-) diff --git a/doc/development/testing_guide/end_to_end_tests.md b/doc/development/testing_guide/end_to_end_tests.md index 87a21b9f7fc..6be0decd75d 100644 --- a/doc/development/testing_guide/end_to_end_tests.md +++ b/doc/development/testing_guide/end_to_end_tests.md @@ -20,23 +20,56 @@ You can find these nightly pipelines at [GitLab QA pipelines page][gitlab-qa-pip ### Testing code in merge requests -It is also possible to trigger packages build and [GitLab QA pipeline][gitlab-qa-pipelines] -using a manual action that should be present in the merge request widget on -your merge request. Look for `package-qa` manual action. +It is also possible to trigger build of GitLab packages and then pass these +package to GitLab QA to run tests in a [pipeline][gitlab-qa-pipelines]. + +Developers can trigger a `package-qa` manual action, that should be present in +the merge request widget in your merge request. + +It is possible to trigger Gitlab QA pipeline from merge requests in GitLab CE +and GitLab EE, but QA triggering manual action is also available in the Omnibus +GitLab project as well. Below you can read more about how to use it and how does it work. -## How does it work? +#### How does it work? -We are using _multi-project pipelines_ to run end-to-end tests. +Currently, we are _multi-project pipeline_-like approach to run QA pipelines. -## How do I test my code? +1. Developer triggers manual action in the CE or EE merge request, that starts +a chain of pipelines. +1. Triggering this action enqueues a new CI job that is going to be picked by a +runner. +1. The script, that is being executed, triggers a pipeline in GitLab Omnibus +projects, and waits for the resulting status. We call it _status attribution_. +1. GitLab packages are being built in the pipeline started in Omnibus. Packages +are going to be sent to Container Registry. +1. When packages are ready, and available in the registry, a final step in the +pipeline that is now running in Omnibus triggers a new pipeline in the GitLab +QA project. It also waits for the resulting status. +1. GitLab QA pulls images from the registry, spins-up containers and runs tests +against test environment that has been just orchestrated. +1. The result of GitLab QA pipeline is being propagated upstream, through +Omnibus, to CE / EE merge request. -## How do I contribute? +#### How do I write tests? + +In order to write new tests, you first need to learn more about GitLab QA +architecture. There is some documentation about it in GitLab QA project +[here][gitlab-qa-architecture]. + +Once you decided we to put test environment orchestration scenarios and +instance specs, take a looks at [relevant documentation][instance-qa-readme] ## Where can I ask for help? +You can ask question in `#qa` channel on Slack (GitLab internal) or you can +find an issue you would like to work on in [the issue tracker][gitlab-qa-issues] +and start a new discussion there. [omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab [gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa [gitlab-qa-pipelines]: https://gitlab.com/gitlab-org/gitlab-qa/pipelines +[instance-qa-readme]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/README.md +[gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md +[gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 4ca192aee7e..74d09eb91ff 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -67,8 +67,8 @@ Everything you should know about how to test Rake tasks. ## [End-to-end tests](end_to_end_tests.md) -Everything you should know about how to run end-to-end tests, also known as -[GitLab QA][gitlab-qa] tests. +Everything you should know about how to run end-to-end tests using +[GitLab QA][gitlab-qa] testing framework. --- From 51bb5abe269c8639b4f9ebbb6cb5e917d2710210 Mon Sep 17 00:00:00 2001 From: Grzegorz Bizon Date: Tue, 2 Jan 2018 14:30:29 +0100 Subject: [PATCH 3/7] Link to end to end test guideline from test pyramid --- doc/development/testing_guide/testing_levels.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 1cbd4350284..dd0e9a77164 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -121,6 +121,9 @@ running feature tests (i.e. using Capybara) against it. The actual test scenarios and steps are [part of GitLab Rails] so that they're always in-sync with the codebase. +Read a separate document about [end-to-end tests](../end_to_end_tests.md) to +learn more. + [multiple pieces]: ../architecture.md#components [GitLab Shell]: https://gitlab.com/gitlab-org/gitlab-shell [GitLab Workhorse]: https://gitlab.com/gitlab-org/gitlab-workhorse From 1aa25a3fe3b7167c2d9c73930cf9a05794be0dd2 Mon Sep 17 00:00:00 2001 From: Grzegorz Bizon Date: Tue, 2 Jan 2018 14:37:01 +0100 Subject: [PATCH 4/7] Copy-edit end-to-end testing guidelines --- .../testing_guide/end_to_end_tests.md | 31 +++++++++++-------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/doc/development/testing_guide/end_to_end_tests.md b/doc/development/testing_guide/end_to_end_tests.md index 6be0decd75d..561547e1581 100644 --- a/doc/development/testing_guide/end_to_end_tests.md +++ b/doc/development/testing_guide/end_to_end_tests.md @@ -36,21 +36,24 @@ Below you can read more about how to use it and how does it work. Currently, we are _multi-project pipeline_-like approach to run QA pipelines. -1. Developer triggers manual action in the CE or EE merge request, that starts -a chain of pipelines. -1. Triggering this action enqueues a new CI job that is going to be picked by a -runner. +1. Developer triggers a manual action, that can be found in CE and EE merge +requests, what starts a chain of pipelines. + 1. The script, that is being executed, triggers a pipeline in GitLab Omnibus -projects, and waits for the resulting status. We call it _status attribution_. -1. GitLab packages are being built in the pipeline started in Omnibus. Packages -are going to be sent to Container Registry. +projects, and waits for the resulting status. We call this a _status attribution_. + +1. GitLab packages are being built in Omnibus pipeline. Packages are going to be +pushed to Container Registry. + 1. When packages are ready, and available in the registry, a final step in the -pipeline that is now running in Omnibus triggers a new pipeline in the GitLab -QA project. It also waits for the resulting status. +pipeline, that is now running in Omnibus, triggers a new pipeline in the GitLab +QA project. It also waits for a resulting status. + 1. GitLab QA pulls images from the registry, spins-up containers and runs tests -against test environment that has been just orchestrated. +against a test environment that has been just orchestrated by `gitlab-qa` tool. + 1. The result of GitLab QA pipeline is being propagated upstream, through -Omnibus, to CE / EE merge request. +Omnibus, back to CE / EE merge request. #### How do I write tests? @@ -58,8 +61,9 @@ In order to write new tests, you first need to learn more about GitLab QA architecture. There is some documentation about it in GitLab QA project [here][gitlab-qa-architecture]. -Once you decided we to put test environment orchestration scenarios and +Once you decided were to put test environment orchestration scenarios and instance specs, take a looks at [relevant documentation][instance-qa-readme] +and examples in [the `qa/` directory][instance-qa-examples]. ## Where can I ask for help? @@ -70,6 +74,7 @@ and start a new discussion there. [omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab [gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa [gitlab-qa-pipelines]: https://gitlab.com/gitlab-org/gitlab-qa/pipelines -[instance-qa-readme]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/README.md [gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md [gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues +[instance-qa-readme]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/README.md +[instance-qa-examples]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa From 124ffb2134be85575ddc75bfa34903fb738c6930 Mon Sep 17 00:00:00 2001 From: Grzegorz Bizon Date: Tue, 2 Jan 2018 14:42:28 +0100 Subject: [PATCH 5/7] Fix link to end-to-end testing docs from test pyramid --- doc/development/testing_guide/testing_levels.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index dd0e9a77164..4adf0dc7c7a 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -121,7 +121,7 @@ running feature tests (i.e. using Capybara) against it. The actual test scenarios and steps are [part of GitLab Rails] so that they're always in-sync with the codebase. -Read a separate document about [end-to-end tests](../end_to_end_tests.md) to +Read a separate document about [end-to-end tests](end_to_end_tests.md) to learn more. [multiple pieces]: ../architecture.md#components From e8acb3f11755811fca28d38bb0cbba44add7b0af Mon Sep 17 00:00:00 2001 From: Grzegorz Bizon Date: Thu, 4 Jan 2018 12:09:14 +0100 Subject: [PATCH 6/7] Copy-edit end-to-end testing guide documentation --- .../testing_guide/end_to_end_tests.md | 34 ++++++++++--------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/doc/development/testing_guide/end_to_end_tests.md b/doc/development/testing_guide/end_to_end_tests.md index 561547e1581..30efe3e3b76 100644 --- a/doc/development/testing_guide/end_to_end_tests.md +++ b/doc/development/testing_guide/end_to_end_tests.md @@ -23,24 +23,25 @@ You can find these nightly pipelines at [GitLab QA pipelines page][gitlab-qa-pip It is also possible to trigger build of GitLab packages and then pass these package to GitLab QA to run tests in a [pipeline][gitlab-qa-pipelines]. -Developers can trigger a `package-qa` manual action, that should be present in -the merge request widget in your merge request. +Developers can trigger the `package-qa` manual action, that should be present in +the merge request widget. -It is possible to trigger Gitlab QA pipeline from merge requests in GitLab CE -and GitLab EE, but QA triggering manual action is also available in the Omnibus -GitLab project as well. +It is also possible to trigger Gitlab QA pipeline from merge requests in +Omnibus GitLab project. You can find a manual action that is similar to +`package-qa`, mentioned above, in your Omnibus-related merge requests as well. Below you can read more about how to use it and how does it work. #### How does it work? -Currently, we are _multi-project pipeline_-like approach to run QA pipelines. +Currently, we are using _multi-project pipeline_-like approach to run QA +pipelines. 1. Developer triggers a manual action, that can be found in CE and EE merge -requests, what starts a chain of pipelines. +requests. This starts a chain of pipelines in multiple projects. -1. The script, that is being executed, triggers a pipeline in GitLab Omnibus -projects, and waits for the resulting status. We call this a _status attribution_. +1. The script being executed triggers a pipeline in GitLab Omnibus and waits +for the resulting status. We call this a _status attribution_. 1. GitLab packages are being built in Omnibus pipeline. Packages are going to be pushed to Container Registry. @@ -50,24 +51,25 @@ pipeline, that is now running in Omnibus, triggers a new pipeline in the GitLab QA project. It also waits for a resulting status. 1. GitLab QA pulls images from the registry, spins-up containers and runs tests -against a test environment that has been just orchestrated by `gitlab-qa` tool. +against a test environment that has been just orchestrated by the `gitlab-qa` +tool. -1. The result of GitLab QA pipeline is being propagated upstream, through +1. The result of the GitLab QA pipeline is being propagated upstream, through Omnibus, back to CE / EE merge request. #### How do I write tests? In order to write new tests, you first need to learn more about GitLab QA -architecture. There is some documentation about it in GitLab QA project -[here][gitlab-qa-architecture]. +architecture. See the [documentation about it][gitlab-qa-architecture] in +GitLab QA project. -Once you decided were to put test environment orchestration scenarios and -instance specs, take a looks at [relevant documentation][instance-qa-readme] +Once you decided where to put test environment orchestration scenarios and +instance specs, take a look at the [relevant documentation][instance-qa-readme] and examples in [the `qa/` directory][instance-qa-examples]. ## Where can I ask for help? -You can ask question in `#qa` channel on Slack (GitLab internal) or you can +You can ask question in the `#qa` channel on Slack (GitLab internal) or you can find an issue you would like to work on in [the issue tracker][gitlab-qa-issues] and start a new discussion there. From 8bdc6c74e82445048d66f6bf4be9dd0db7dc4737 Mon Sep 17 00:00:00 2001 From: Grzegorz Bizon Date: Fri, 5 Jan 2018 15:32:41 +0100 Subject: [PATCH 7/7] Rephrase paragraph about e2e tests in merge requests in docs --- doc/development/testing_guide/end_to_end_tests.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/doc/development/testing_guide/end_to_end_tests.md b/doc/development/testing_guide/end_to_end_tests.md index 30efe3e3b76..abe5b06e0f0 100644 --- a/doc/development/testing_guide/end_to_end_tests.md +++ b/doc/development/testing_guide/end_to_end_tests.md @@ -20,15 +20,13 @@ You can find these nightly pipelines at [GitLab QA pipelines page][gitlab-qa-pip ### Testing code in merge requests -It is also possible to trigger build of GitLab packages and then pass these -package to GitLab QA to run tests in a [pipeline][gitlab-qa-pipelines]. +It is possible to run end-to-end tests (eventually being run within a +[GitLab QA pipeline][gitlab-qa-pipelines]) for a merge request by triggering +the `package-qa` manual action, that should be present in a merge request +widget. -Developers can trigger the `package-qa` manual action, that should be present in -the merge request widget. - -It is also possible to trigger Gitlab QA pipeline from merge requests in -Omnibus GitLab project. You can find a manual action that is similar to -`package-qa`, mentioned above, in your Omnibus-related merge requests as well. +Mmanual action that starts end-to-end tests is also available in merge requests +in Omnibus GitLab project. Below you can read more about how to use it and how does it work.