diff --git a/doc/README.md b/doc/README.md index a814c787f94..4248f62c08c 100644 --- a/doc/README.md +++ b/doc/README.md @@ -133,6 +133,7 @@ scales to run your tests faster. - [GitLab CI/CD](ci/README.md): Explore the features and capabilities of Continuous Integration, Continuous Delivery, and Continuous Deployment with GitLab. - [Review Apps](ci/review_apps/index.md): Preview changes to your app right from a merge request. - [Pipeline Graphs](ci/pipelines.md#pipeline-graphs) +- [JUnit test reports](ci/junit_test_reports.md) ### Package diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 811f4d1f07a..8eb96ae10b2 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -43,6 +43,10 @@ There's also a collection of repositories with [example projects](https://gitlab - [Using `dpl` as deployment tool](deployment/README.md) - [The `.gitlab-ci.yml` file for GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml) +## Test Reports + +[Collect test reports in Verify stage](../junit_test_reports.md). + ## Code Quality analysis **(Starter)** [Analyze your project's Code Quality](code_quality.md). diff --git a/doc/ci/img/junit_test_report.png b/doc/ci/img/junit_test_report.png new file mode 100644 index 00000000000..ad098eb457f Binary files /dev/null and b/doc/ci/img/junit_test_report.png differ diff --git a/doc/ci/junit_test_reports.md b/doc/ci/junit_test_reports.md new file mode 100644 index 00000000000..5ae8ecaafa6 --- /dev/null +++ b/doc/ci/junit_test_reports.md @@ -0,0 +1,102 @@ +# JUnit test reports + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/45318) in GitLab 11.2. +Requires GitLab Runner 11.2 and above. + +## Overview + +It is very common that a [CI/CD pipeline](pipelines.md) contains a +test job that will verify your code. +If the tests fail, the pipeline fails and users get notified. The person that +works on the merge request will have to check the job logs and see where the +tests failed so that they can fix them. + +You can configure your job to use JUnit test reports, and GitLab will display a +report on the merge request so that it's easier and faster to identify the +failure without having to check the entire log. + +## Use cases + +Consider the following workflow: + +1. Your `master` branch is rock solid, your project is using GitLab CI/CD and + your pipelines indicate that there isn't anything broken. +1. Someone from you team submits a merge request, a test fails and the pipeline + gets the known red icon. To investigate more, you have to go through the job + logs to figure out the cause of the failed test, which usually contain + thousands of lines. +1. You configure the JUnit test reports and immediately GitLab collects and + exposes them in the merge request. No more searching in the job logs. +1. Your development and debugging workflow becomes easier, faster and efficient. + +## How it works + +First, GitLab Runner uploads all JUnit XML files as artifacts to GitLab. Then, +when you visit a merge request, GitLab starts comparing the head and base branch's +JUnit test reports, where: + +- The base branch is the target branch (usually `master`). +- The head branch is the source branch (the latest pipeline in each merge request). + +The reports panel has a summary showing how many tests failed and how many were fixed. +If no comparison can be done because data for the base branch is not available, +the panel will just show the list of failed tests for head. + +There are three types of results: + +1. **Newly failed tests:** Test cases which passed on base branch and failed on head branch +1. **Existing failures:** Test cases which failed on base branch and failed on head branch +1. **Resolved failures:** Test cases which failed on base branch and passed on head branch + +Each entry in the panel will show the test name and its type from the list +above. Clicking on the test name will open a modal window with details of its +execution time and the error output. + +![Test Reports Widget](img/junit_test_report.png) + +## How to set it up + +NOTE: **Note:** +For a list of supported languages on JUnit tests, check the +[Wikipedia article](https://en.wikipedia.org/wiki/JUnit#Ports). + +To enable the JUnit reports in merge requests, you need to add +[`artifacts:reports:junit`](yaml/README.md#artifacts-reports-junit) +in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. + +In the following examples, the job in the `test` stage runs and GitLab +collects the JUnit test report from each job. After each job is executed, the +XML reports are stored in GitLab as artifacts and their results are shown in the +merge request widget. + +### Ruby example + +Use the following job in `.gitlab-ci.yml`: + +```yaml +## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report with rspec +ruby: + stage: test + script: + - bundle install + - rspec spec/lib/ --format RspecJunitFormatter --out rspec.xml + artifacts: + reports: + junit: rspec.xml +``` + +### Go example + +Use the following job in `.gitlab-ci.yml`: + +```yaml +## Use https://github.com/jstemmer/go-junit-report to generate a JUnit report with go +golang: + stage: test + script: + - go get -u github.com/jstemmer/go-junit-report + - go test -v 2>&1 | go-junit-report > report.xml + artifacts: + reports: + junit: report.xml +``` diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 95d705d3a3d..ef740ab1c5e 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -1092,6 +1092,52 @@ job: expire_in: 1 week ``` +### `artifacts:reports` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20390) in +GitLab 11.2. Requires GitLab Runner 11.2 and above. + +The `reports` keyword is used for collecting test reports from jobs and +exposing them in GitLab's UI (merge requests, pipeline views). Read how to use +this with [JUnit reports](#artifacts-reports-junit). + +NOTE: **Note:** +The test reports are collected regardless of the job results (success or failure). +You can use [`artifacts:expire_in`](#artifacts-expire_in) to set up an expiration +date for their artifacts. + +#### `artifacts:reports:junit` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20390) in +GitLab 11.2. Requires GitLab Runner 11.2 and above. + +The `junit` report collects [JUnit XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) +as artifacts. Although JUnit was originally developed in Java, there are many +[third party ports](https://en.wikipedia.org/wiki/JUnit#Ports) for other +languages like Javascript, Python, Ruby, etc. + +Below is an example of collecting a JUnit XML file from Ruby's RSpec test tool: + +```yaml +rspec: + stage: test + script: + - bundle install + - rspec --format RspecJunitFormatter --out rspec.xml + artifacts: + reports: + junit: rspec.xml +``` + +The collected JUnit reports will be uploaded to GitLab as an artifact and will +be automatically [shown in merge requests](../junit_test_reports.md). + +NOTE: **Note:** +In case the JUnit tool you use exports to multiple XML files, you can specify +multiple test report paths within a single job +(`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`) and they will be automatically +concatenated into a single file. + ## `dependencies` > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 86ecf33ed31..43ca498d006 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -43,8 +43,7 @@ A. Consider you are a software developer working in a team: 1. You checkout a new branch, and submit your changes through a merge request 1. You gather feedback from your team -1. You work on the implementation optimizing code with [Code Quality reports](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html) **[STARTER]** -1. You build and test your changes with GitLab CI/CD +1. You verify your changes with [JUnit test reports](../../../ci/junit_test_reports.md) in GitLab CI/CD 1. You request the approval from your manager 1. Your manager pushes a commit with his final review, [approves the merge request](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter) 1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#manual-actions) for GitLab CI/CD