GitLab tests in the Continuous Integration (CI) context

Our current CI parallelization setup is as follows:

The knapsack job in the prepare stage that is supposed to ensure we have a knapsack/${CI_PROJECT_NAME}/rspec_report-master.json file:
- The knapsack/${CI_PROJECT_NAME}/rspec_report-master.json file is fetched from S3, if it's not here we initialize the file with {}.
Each rspec x y job are run with knapsack rspec and should have an evenly distributed share of tests:
- It works because the jobs have access to the knapsack/${CI_PROJECT_NAME}/rspec_report-master.json since the "artifacts from all previous stages are passed by default".
- the jobs set their own report path to KNAPSACK_REPORT_PATH=knapsack/${CI_PROJECT_NAME}/${JOB_NAME[0]}_node_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json.
- if knapsack is doing its job, test files that are run should be listed under Report specs, not under Leftover specs.
The update-knapsack job takes all the knapsack/${CI_PROJECT_NAME}/${JOB_NAME[0]}_node_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json files from the rspec x y jobs and merge them all together into a single knapsack/${CI_PROJECT_NAME}/rspec_report-master.json file that is then uploaded to S3.

After that, the next pipeline will use the up-to-date knapsack/${CI_PROJECT_NAME}/rspec_report-master.json file.

The GitLab test suite is monitored for the master branch, and any branch that includes rspec-profile in their name.

A public dashboard is available for everyone to see. Feel free to look at the slowest test files and try to improve them.

CI setup

On CE and EE, the test suite runs both PostgreSQL and MySQL.
Rails logging to log/test.log is disabled by default in CI for performance reasons. To override this setting, provide the RAILS_ENABLE_TEST_LOG environment variable.