2019-02-19 21:44:08 +00:00
# GitLab QA - End-to-end tests for GitLab
2017-02-20 11:45:04 +00:00
2019-02-19 21:44:08 +00:00
This directory contains [end-to-end tests ](doc/development/testing_guide/end_to_end_tests.md )
for GitLab. It includes the test framework and the tests themselves.
The tests can be found in `qa/specs/features` (not to be confused with the unit
tests for the test framework, which are in `spec/` ).
2017-02-20 12:18:41 +00:00
2017-10-05 10:36:28 +00:00
It is part of the [GitLab QA project ](https://gitlab.com/gitlab-org/gitlab-qa ).
2017-03-09 09:41:10 +00:00
2017-10-05 10:36:28 +00:00
## What is it?
2017-03-09 09:41:10 +00:00
2019-02-19 21:44:08 +00:00
GitLab QA is an end-to-end tests suite for GitLab.
2017-03-09 09:41:10 +00:00
2019-02-19 21:44:08 +00:00
These are black-box and entirely click-driven end-to-end tests you can run
2017-03-09 09:41:10 +00:00
against any existing instance.
## How does it work?
1. When we release a new version of GitLab, we build a Docker images for it.
1. Along with GitLab Docker Images we also build and publish GitLab QA images.
2019-02-19 21:44:08 +00:00
1. GitLab QA project uses these images to execute end-to-end tests.
2017-09-30 06:38:12 +00:00
2018-01-12 10:04:41 +00:00
## Validating GitLab views / partials / selectors in merge requests
We recently added a new CI job that is going to be triggered for every push
event in CE and EE projects. The job is called `qa:selectors` and it will
verify coupling between page objects implemented as a part of GitLab QA
and corresponding views / partials / selectors in CE / EE.
Whenever `qa:selectors` job fails in your merge request, you are supposed to
fix [page objects ](qa/page/README.md ). You should also trigger end-to-end tests
2018-03-19 09:04:23 +00:00
using `package-and-qa` manual action, to test if everything works fine.
2018-01-12 10:04:41 +00:00
2017-09-30 06:38:12 +00:00
## How can I use it?
You can use GitLab QA to exercise tests on any live instance! For example, the
2017-10-05 10:36:28 +00:00
following call would login to a local [GDK] instance and run all specs in
2017-09-30 06:38:12 +00:00
`qa/specs/features` :
```
2018-08-20 18:13:54 +00:00
bin/qa Test::Instance::All http://localhost:3000
2017-09-30 06:38:12 +00:00
```
2019-02-28 19:30:04 +00:00
Note: If you want to run tests requiring SSH against GDK, you
will need to [modify your GDK setup ](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/run_qa_against_gdk.md ).
2018-01-11 09:26:48 +00:00
### Writing tests
1. [Using page objects ](qa/page/README.md )
2017-10-05 10:36:28 +00:00
### Running specific tests
You can also supply specific tests to run as another parameter. For example, to
2018-01-11 09:26:48 +00:00
run the repository-related specs, you can execute:
2017-09-30 06:38:12 +00:00
```
2018-08-20 18:13:54 +00:00
bin/qa Test::Instance::All http://localhost qa/specs/features/repository/
2017-12-20 11:03:10 +00:00
```
Since the arguments would be passed to `rspec` , you could use all `rspec`
options there. For example, passing `--backtrace` and also line number:
```
2018-08-20 18:13:54 +00:00
bin/qa Test::Instance::All http://localhost qa/specs/features/project/create_spec.rb:3 --backtrace
2017-10-05 10:36:28 +00:00
```
### Overriding the authenticated user
Unless told otherwise, the QA tests will run as the default `root` user seeded
by the GDK.
If you need to authenticate as a different user, you can provide the
`GITLAB_USERNAME` and `GITLAB_PASSWORD` environment variables:
```
2018-08-20 18:13:54 +00:00
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bin/qa Test::Instance::All https://gitlab.example.com
2017-09-30 06:38:12 +00:00
```
2018-02-08 14:34:56 +00:00
If your user doesn't have permission to default sandbox group
`gitlab-qa-sandbox` , you could also use another sandbox group by giving
`GITLAB_SANDBOX_NAME` :
```
2018-08-20 18:13:54 +00:00
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance::All https://gitlab.example.com
2018-02-08 14:34:56 +00:00
```
2018-09-10 19:01:55 +00:00
All [supported environment variables are here ](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-environment-variables ).
2017-10-05 10:36:28 +00:00
2018-11-28 13:11:05 +00:00
### Sending additional cookies
The environment variable `QA_COOKIES` can be set to send additional cookies
on every request. This is necessary on gitlab.com to direct traffic to the
canary fleet. To do this set `QA_COOKIES="gitlab_canary=true"` .
2018-12-28 16:06:08 +00:00
To set multiple cookies, separate them with the `;` character, for example: `QA_COOKIES="cookie1=value;cookie2=value2"`
2018-11-28 13:11:05 +00:00
2018-02-09 16:49:10 +00:00
### Building a Docker image to test
Once you have made changes to the CE/EE repositories, you may want to build a
Docker image to test locally instead of waiting for the `gitlab-ce-qa` or
`gitlab-ee-qa` nightly builds. To do that, you can run from this directory:
```sh
docker build -t gitlab/gitlab-ce-qa:nightly .
```
2017-10-05 10:36:28 +00:00
[GDK]: https://gitlab.com/gitlab-org/gitlab-development-kit/
2018-12-28 16:06:08 +00:00
### Quarantined tests
Tests can be put in quarantine by assigning `:quarantine` metadata. This means
they will be skipped unless run with `--tag quarantine` . This can be used for
tests that are expected to fail while a fix is in progress (similar to how
[`skip` or `pending` ](https://relishapp.com/rspec/rspec-core/v/3-8/docs/pending-and-skipped-examples )
can be used).
```
bin/qa Test::Instance::All http://localhost --tag quarantine
```
If `quarantine` is used with other tags, tests will only be run if they have at
least one of the tags other than `quarantine` . This is different from how RSpec
tags usually work, where all tags are inclusive.
For example, suppose one test has `:smoke` and `:quarantine` metadata, and
another test has `:ldap` and `:quarantine` metadata. If the tests are run with
`--tag smoke --tag quarantine` , only the first test will run. The test with
`:ldap` will not run even though it also has `:quarantine` .