kotovalexarian-likes-gitlab
/
gitlab-org--gitlab-foss
1
0
Fork 0
Code Releases Activity

Add latest changes from gitlab-org/gitlab@master

This commit is contained in:
GitLab Bot 2022-10-19 03:10:36 +00:00
parent 18e11db5dc
commit 53181fac03
19 changed files with 312 additions and 76 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
GITALY_SERVER_VERSION
View File

@ -1 +1 @@
1c2e1293595d2a28690e8e9377b246663b5380c0
334a620a54df6bbb1563c440514e06d7068255e7

2
Gemfile
View File

@ -512,7 +512,7 @@ gem 'grpc', '~> 1.42.0'
gem 'google-protobuf', '~> 3.21'
gem 'toml-rb', '~> 2.0'
gem 'toml-rb', '~> 2.2.0'
# Feature toggles
gem 'flipper', '~> 0.25.0'

2
Gemfile.checksum
View File

@ -585,7 +585,7 @@
{"name":"timeliness","version":"0.3.10","platform":"ruby","checksum":"c357233ce19dc53148e8b29dfddde134689f18f52b32928e9dfe12ebcf4a773f"},
{"name":"timfel-krb5-auth","version":"0.8.3","platform":"ruby","checksum":"ab388c9d747fa3cd95baf2cc1c03253e372d8c680adcc543670f4f099854bb80"},
{"name":"tins","version":"1.31.1","platform":"ruby","checksum":"51c4a347c25c630d310cbc2c040ffb84e266c8227f2ade881f1130ee4f9fbecf"},
{"name":"toml-rb","version":"2.0.1","platform":"ruby","checksum":"5016c6c77ac72bca5fe67c372722bdfdd4479a6fe1a1c4ff2a486e247849b274"},
{"name":"toml-rb","version":"2.2.0","platform":"ruby","checksum":"a1e2c54ac3cc9d49861004f75f0648b3622ac03a76abe105358c31553227d9a6"},
{"name":"tomlrb","version":"1.3.0","platform":"ruby","checksum":"68666bf53fa70ba686a48a7435ce7e086f5227c58c4c993bd9792f4760f2a503"},
{"name":"tpm-key_attestation","version":"0.9.0","platform":"ruby","checksum":"e469ad9111a68dab4d04596e1c0621d7c877c2e3e247f765af3c04f1adf2b8cd"},
{"name":"train-core","version":"3.4.9","platform":"ruby","checksum":"d7ad8fa9a379c43a30baaaf1141af1cb28349d386c054f7fc81d169a625d6edd"},

4
Gemfile.lock
View File

@ -1417,7 +1417,7 @@ GEM
timfel-krb5-auth (0.8.3)
tins (1.31.1)
sync
toml-rb (2.0.1)
toml-rb (2.2.0)
citrus (~> 3.0, > 3.0)
tomlrb (1.3.0)
tpm-key_attestation (0.9.0)
@ -1813,7 +1813,7 @@ DEPENDENCIES
thrift (>= 0.16.0)
timecop (~> 0.9.1)
timfel-krb5-auth (~> 0.8)
toml-rb (~> 2.0)
toml-rb (~> 2.2.0)
truncato (~> 0.7.12)
typhoeus (~> 1.4.0)
u2f (~> 0.2.1)

7
config/dependency_decisions.yml
View File

@ -362,3 +362,10 @@
:why: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79864#note_845406018
:versions: [2.0.1]
:when: 2022-02-24 10:44:26.669326000 Z
- - :license
- gridstack
- MIT
- :who: Balasankar C
:why: https://github.com/gridstack/gridstack.js/blob/v7.0.0/LICENSE
:versions: []
:when: 2022-10-18 16:24:56.611523399 Z

185
doc/development/sec/analyzer_development_guide.md Normal file
View File

@ -0,0 +1,185 @@
---
stage: Secure
group: Static Analysis
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Sec section analyzer development
Analyzers are shipped as Docker images to execute within a CI pipeline context. This guide describes development and testing
practices across analyzers.
## Shared modules
There are a number of shared Go modules shared across analyzers for common behavior and interfaces:
- The [`command`](https://gitlab.com/gitlab-org/security-products/analyzers/command#how-to-use-the-library) Go package implements a CLI interface.
- The [`common`](https://gitlab.com/gitlab-org/security-products/analyzers/common) project provides miscellaneous shared modules for logging, certificate handling, and directory search capabilities.
- The [`report`](https://gitlab.com/gitlab-org/security-products/analyzers/report) Go package's `Report` and `Finding` structs marshal JSON reports.
- The [`template`](https://gitlab.com/gitlab-org/security-products/analyzers/template) project scaffolds new analyzers.
## How to use the analyzers
Analyzers are shipped as Docker images. For example, to run the
[semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) Docker image to scan the working directory:
1. `cd` into the directory of the source code you want to scan.
1. Run `docker login registry.gitlab.com` and provide username plus
[personal](../../user/profile/personal_access_tokens.md#create-a-personal-access-token)
or [project](../../user/project/settings/project_access_tokens.md#create-a-project-access-token)
access token with at least the `read_registry` scope.
1. Run the Docker image:
```shell
docker run \
--interactive --tty --rm \
--volume "$PWD":/tmp/app \
--env CI_PROJECT_DIR=/tmp/app \
-w /tmp/app \
registry.gitlab.com/gitlab-org/security-products/analyzers/semgrep:latest /analyzer run
```
1. The Docker container generates a report in the mounted project directory with a report filename corresponding to the analyzer category. For example, [SAST](../../user/application_security/sast) generates a file named `gl-sast-report.json`.
## Analyzers development
To update the analyzer:
1. Modify the Go source code.
1. Build a new Docker image.
1. Run the analyzer against its test project.
1. Compare the generated report with what's expected.
Here's how to create a Docker image named `analyzer`:
```shell
docker build -t analyzer .
```
For example, to test Secret Detection run the following:
```shell
wget https://gitlab.com/gitlab-org/security-products/ci-templates/-/raw/master/scripts/compare_reports.sh
sh ./compare_reports.sh sd test/fixtures/gl-secret-detection-report.json test/expect/gl-secret-detection-report.json \
| patch -Np1 test/expect/gl-secret-detection-report.json && Git commit -m 'Update expectation' test/expect/gl-secret-detection-report.json
rm compare_reports.sh
```
You can also compile the binary for your own environment and run it locally
but `analyze` and `run` probably won't work
since the runtime dependencies of the analyzer are missing.
Here's an example based on
[SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs):
```shell
go build -o analyzer
./analyzer search test/fixtures
./analyzer convert test/fixtures/app/spotbugsXml.Xml > ./gl-sast-report.json
```
## How to test the analyzers
Video walkthrough of how Dependency Scanning analyzers are using [downstream pipeline](../../ci/pipelines/downstream_pipelines.md) feature to test analyzers using test projects:
[![How Sec leverages the downstream pipeline feature of GitLab to test analyzers end to end](http://img.youtube.com/vi/KauRBlfUbDE/0.jpg)](http://www.youtube.com/watch?v=KauRBlfUbDE)
### Testing local changes
To test local changes in the shared modules (such as `command` or `report`) for an analyzer
you can use the
[`go mod replace`](https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive)
directive to load `command` with your local changes instead of using the version of command that has been
tagged remotely. For example:
```shell
go mod edit -replace gitlab.com/gitlab-org/security-products/analyzers/command/v3=/local/path/to/command
```
Alternatively you can achieve the same result by manually updating the `go.mod` file:
```plaintext
module gitlab.com/gitlab-org/security-products/analyzers/awesome-analyzer/v2
replace gitlab.com/gitlab-org/security-products/analyzers/command/v3 => /path/to/command
require (
...
gitlab.com/gitlab-org/security-products/analyzers/command/v3 v2.19.0
)
```
#### Testing local changes in Docker
To use Docker with `replace` in the `go.mod` file:
1. Copy the contents of `command` into the directory of the analyzer. `cp -r /path/to/command path/to/analyzer/command`.
1. Add a copy statement in the analyzer's `Dockerfile`: `COPY command /command`.
1. Update the `replace` statement to make sure it matches the destination of the `COPY` statement in the step above:
`replace gitlab.com/gitlab-org/security-products/analyzers/command/v3 => /command`
## Versioning and release process
Analyzers are independent projects that follow their own versioning. `Patch` version bumps tend to correspond to a `Minor` version bump of the underlying tools (i.e. [`bandit`](https://wiki.openstack.org/wiki/Security/Projects/Bandit)), allowing us greater flexibility in reserving `Minor` bumps for more significant changes to our scanners. In case of breaking changes imposed by the wrapped scanner, creating a new analyzer on a separate repository must be considered.
The analyzers are released as Docker images following this scheme:
- each push to the `master` branch will override the `edge` image tag
- each push to any `awesome-feature` branch will generate a matching `awesome-feature` image tag
- each Git tag will generate the corresponding `Major.Minor.Patch` image tag. A manual job allows to override the corresponding `Major` and the `latest` image tags to point to this `Major.Minor.Patch`.
To release a new analyzer Docker image, there are two different options:
- Manual release process
- Automatic release process
### Manual release process
1. Ensure that the `CHANGELOG.md` entry for the new analyzer is correct.
1. Ensure that the release source (typically the `master` or `main` branch) has a passing pipeline.
1. Create a new release for the analyzer project by selecting the **Deployments** menu on the left-hand side of the project window, then selecting the **Releases** sub-menu.
1. Select **New release** to open the **New Release** page.
1. In the **Tag name** drop down, enter the same version used in the `CHANGELOG.md`, for example `v2.4.2`, and select the option to create the tag (`Create tag v2.4.2` here).
1. In the **Release title** text box enter the same version used above, for example `v2.4.2`.
1. In the `Release notes` text box, copy and paste the notes from the corresponding version in the `CHANGELOG.md`.
1. Leave all other settings as the default values.
1. Select **Create release**.
After following the above process and creating a new release, a new Git tag is created with the `Tag name` provided above. This triggers a new pipeline with the given tag version and a new analyzer Docker image is built.
If the analyzer uses the [`analyzer.yml` template](https://gitlab.com/gitlab-org/security-products/ci-templates/blob/b446fd3/includes-dev/analyzer.yml#L209-217), then the pipeline triggered as part of the **New release** process above automatically tags and deploys a new version of the analyzer Docker image.
If the analyzer does not use the `analyzer.yml` template, you'll need to manually tag and deploy a new version of the analyzer Docker image:
1. Select the **CI/CD** menu on the left-hand side of the project window, then select the **Pipelines** sub-menu.
1. A new pipeline should currently be running with the same tag used previously, for example `v2.4.2`.
1. After the pipeline has completed, it will be in a `blocked` state.
1. Select the `Manual job` play button on the right hand side of the window and select `tag version` to tag and deploy a new version of the analyzer Docker image.
Use your best judgment to decide when to create a Git tag, which will then trigger the release job. If you
can't decide, then ask for other's input.
### Automatic release process
The following must be performed before the automatic release process can be used:
1. Configure `CREATE_GIT_TAG: true` as a [`CI/CD` environment variable](../../ci/variables/index.md).
1. Check the `Variables` in the CI/CD project settings. Unless the project already inherits the `GITLAB_TOKEN` environment variable from the project group, create a [project access token](../../user/project/settings/project_access_tokens.md) with `complete read/write access to the API` and configure `GITLAB_TOKEN` as a [`CI/CD` environment variable](../../ci/variables/index.md) which refers to this token.
After the above steps have been completed, the automatic release process executes as follows:
1. A project maintainer merges an MR into the default branch.
1. The default pipeline is triggered, and the `upsert git tag` job is executed.
- If the most recent version in the `CHANGELOG.md` matches one of the Git tags, the job is a no-op.
- Else, this job automatically creates a new release and Git tag using the [releases API](../../api/releases/index.md#create-a-release). The version and message is obtained from the most recent entry in the `CHANGELOG.md` file for the project.
1. A pipeline is automatically triggered for the new Git tag. This pipeline releases the `latest`, `major`, `minor` and `patch` Docker images of the analyzer.
### Steps to perform after releasing an analyzer
1. After a new version of the analyzer Docker image has been tagged and deployed, please test it with the corresponding test project.
1. Announce the release on the relevant group Slack channel. Example message:
> FYI I've just released `ANALYZER_NAME` `ANALYZER_VERSION`. `LINK_TO_RELEASE`
**Never delete a Git tag that has been pushed** as there is a good
chance that the tag will be used and/or cached by the Go package registry.

3
doc/development/sec/index.md
View File

@ -18,6 +18,7 @@ See [Terminology](../../user/application_security/terminology) for an overview o
- [Scanning](#scanning)
- [Processing, visualization, and management](#processing-visualization-and-management)
- [Severity Levels](../../user/application_security/vulnerabilities/severities.md)
- [Analyzer Development](analyzer_development_guide.md)
## Overview
@ -65,5 +66,5 @@ Depending on the context, the security reports may be stored either in the datab
## CI/CD template development
While CI/CD templates are the responsibiility of the Verify section, many are critical to the Sec Section's feature usage.
While CI/CD templates are the responsibility of the Verify section, many are critical to the Sec Section's feature usage.
If you are working with CI/CD templates, please read the [development guide for GitLab CI/CD templates](../cicd/templates.md).

11
doc/user/application_security/index.md
View File

@ -175,16 +175,21 @@ the instructions for
By default, the application security jobs are configured to run for branch pipelines only.
To use them with [merge request pipelines](../../ci/pipelines/merge_request_pipelines.md),
you must reference the [`latest` templates](../../development/cicd/templates.md):
you must reference the [`latest` templates](../../development/cicd/templates.md).
All `latest` security templates support merge request pipelines.
For example, to run both SAST and Dependency Scanning:
```yaml
include:
- template: Security/Container-Scanning.latest.gitlab-ci.yml
- template: Security/DAST.latest.gitlab-ci.yml
- template: Jobs/Dependency-Scanning.latest.gitlab-ci.yml
- template: Jobs/SAST.latest.gitlab-ci.yml
```
NOTE:
Mixing `latest` and `stable` security templates can cause both MR and branch pipelines to run. We recommend choosing `latest` or `stable` for all security scanners.
NOTE:
Latest templates can receive breaking changes in any release.

11
lib/banzai/filter/wiki_link_filter/rewriter.rb
View File

@ -42,7 +42,7 @@ module Banzai
def apply_relative_link_rules!
if @uri.relative? && @uri.path.present?
link = @uri.path
link = ::File.join(@wiki_base_path, link) unless link.starts_with?(@wiki_base_path)
link = ::File.join(@wiki_base_path, link) unless prefixed_with_base_path?(link)
link = "#{link}##{@uri.fragment}" if @uri.fragment
@uri = Addressable::URI.parse(link)
end
@ -55,6 +55,15 @@ module Banzai
def repository_upload?
@uri.relative? && @uri.path.starts_with?(Wikis::CreateAttachmentService::ATTACHMENT_PATH)
end
def prefixed_with_base_path?(link)
link.starts_with?(@wiki_base_path) || link.starts_with?(old_wiki_base_path)
end
# before we added `/-/` to all our paths
def old_wiki_base_path
@wiki_base_path.sub('/-/', '/')
end
end
end
end

21
locale/gitlab.pot
View File

@ -3952,7 +3952,7 @@ msgstr ""
msgid "All users with matching cards"
msgstr ""
msgid "Allow \"%{group_name}\" to sign you in"
msgid "Allow %{strongOpen}%{group_name}%{strongClose} to sign you in?"
msgstr ""
msgid "Allow access to members of the following group"
@ -35303,16 +35303,22 @@ msgstr ""
msgid "SAML for %{group_name}"
msgstr ""
msgid "SAML|Select \"Authorize\" to allow %{group_name} to manage your GitLab account \"%{username}\" (%{email}) after you sign in successfully with your single sign-on account."
msgid "SAML single sign-on"
msgstr ""
msgid "SAML single sign-on for %{group_name}"
msgstr ""
msgid "SAML|Sign in to GitLab to connect your organization's account"
msgstr ""
msgid "SAML|The \"%{group_path}\" group allows you to sign in with your Single Sign-On Account."
msgid "SAML|The %{strongOpen}%{group_path}%{strongClose} group allows you to sign in using single sign-on."
msgstr ""
msgid "SAML|To access \"%{group_name}\" you must sign in with your Single Sign-On account, through an external sign-in page."
msgid "SAML|To access %{strongOpen}%{group_name}%{strongClose}, you must sign in using single sign-on through an external sign-in page."
msgstr ""
msgid "SAML|To allow %{strongOpen}%{group_name}%{strongClose} to manage your GitLab account %{strongOpen}%{username}%{strongClose} (%{email}) after you sign in successfully using single sign-on, select %{strongOpen}Authorize%{strongClose}."
msgstr ""
msgid "SAML|Your organization's SSO has been connected to your GitLab account"
@ -37660,7 +37666,7 @@ msgstr ""
msgid "Sign in preview"
msgstr ""
msgid "Sign in to \"%{group_name}\""
msgid "Sign in to %{group_name}"
msgstr ""
msgid "Sign in to GitLab"
@ -37675,7 +37681,7 @@ msgstr ""
msgid "Sign in with"
msgstr ""
msgid "Sign in with Single Sign-On"
msgid "Sign in with single sign-on"
msgstr ""
msgid "Sign in with smart card"
@ -41276,9 +41282,6 @@ msgstr ""
msgid "This epic cannot be added. An epic cannot be added to itself."
msgstr ""
msgid "This epic cannot be added. An epic cannot belong to an ancestor group of its parent epic."
msgstr ""
msgid "This epic cannot be added. An epic must belong to the same group or subgroup as its parent epic."
msgstr ""

4
qa/Gemfile
View File

@ -8,7 +8,7 @@ gem 'allure-rspec', '~> 2.18.0'
gem 'capybara', '~> 3.35.0'
gem 'capybara-screenshot', '~> 1.0.26'
gem 'rake', '~> 13'
gem 'rspec', '~> 3.10'
gem 'rspec', '~> 3.11'
gem 'selenium-webdriver', '~> 4.5'
gem 'airborne', '~> 0.3.7', require: false # airborne is messing with rspec sandboxed mode so not requiring by default
gem 'rest-client', '~> 2.1.0'
@ -44,6 +44,6 @@ gem 'nokogiri', '~> 1.13', '>= 1.13.8'
gem 'deprecation_toolkit', '~> 2.0.0', require: false
group :development do
gem 'pry-byebug', '~> 3.5.1', platform: :mri
gem 'pry-byebug', '~> 3.10.1', platform: :mri
gem "ruby-debug-ide", "~> 0.7.3"
end

44
qa/Gemfile.lock
View File

@ -26,7 +26,7 @@ GEM
ast (2.4.2)
binding_ninja (0.2.3)
builder (3.2.4)
byebug (9.1.0)
byebug (11.1.3)
capybara (3.35.3)
addressable
mini_mime (>= 0.1.3)
@ -166,7 +166,7 @@ GEM
macaddr (1.7.2)
systemu (~> 2.6.5)
memoist (0.16.2)
method_source (0.9.0)
method_source (1.0.0)
mime-types (3.4.1)
mime-types-data (~> 3.2015)
mime-types-data (3.2022.0105)
@ -193,12 +193,12 @@ GEM
coderay
parser
unparser
pry (0.11.3)
coderay (~> 1.1.0)
method_source (~> 0.9.0)
pry-byebug (3.5.1)
byebug (~> 9.1)
pry (~> 0.10)
pry (0.14.1)
coderay (~> 1.1)
method_source (~> 1.0)
pry-byebug (3.10.1)
byebug (~> 11.0)
pry (>= 0.13, < 0.15)
public_suffix (5.0.0)
racc (1.6.0)
rack (2.2.3.1)
@ -220,27 +220,27 @@ GEM
retriable (3.1.2)
rexml (3.2.5)
rotp (6.2.0)
rspec (3.10.0)
rspec-core (~> 3.10.0)
rspec-expectations (~> 3.10.0)
rspec-mocks (~> 3.10.0)
rspec-core (3.10.1)
rspec-support (~> 3.10.0)
rspec-expectations (3.10.1)
rspec (3.11.0)
rspec-core (~> 3.11.0)
rspec-expectations (~> 3.11.0)
rspec-mocks (~> 3.11.0)
rspec-core (3.11.0)
rspec-support (~> 3.11.0)
rspec-expectations (3.11.1)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.10.0)
rspec-mocks (3.10.2)
rspec-support (~> 3.11.0)
rspec-mocks (3.11.1)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.10.0)
rspec-support (~> 3.11.0)
rspec-parameterized (0.5.2)
binding_ninja (>= 0.2.3)
parser
proc_to_ast
rspec (>= 2.13, < 4)
unparser
rspec-retry (0.6.1)
rspec-retry (0.6.2)
rspec-core (> 3.3)
rspec-support (3.10.2)
rspec-support (3.11.1)
rspec_junit_formatter (0.6.0)
rspec-core (>= 2, < 4, != 2.12.0)
ruby-debug-ide (0.7.3)
@ -317,12 +317,12 @@ DEPENDENCIES
octokit (~> 5.6.1)
parallel (~> 1.19)
parallel_tests (~> 3.13)
pry-byebug (~> 3.5.1)
pry-byebug (~> 3.10.1)
rainbow (~> 3.0.0)
rake (~> 13)
rest-client (~> 2.1.0)
rotp (~> 6.2.0)
rspec (~> 3.10)
rspec (~> 3.11)
rspec-parameterized (~> 0.5.2)
rspec-retry (~> 0.6.1)
rspec_junit_formatter (~> 0.6.0)

4
spec/graphql/gitlab_schema_spec.rb
View File

@ -319,8 +319,8 @@ RSpec.describe GitlabSchema do
subject(:parse_gids) { described_class.parse_gids(global_ids, expected_type: [TestOne, TestTwo]) }
it 'parses the gids' do
expect(described_class).to receive(:parse_gid).with('gid://gitlab/TestOne/123', expected_type: [TestOne, TestTwo]).and_call_original
expect(described_class).to receive(:parse_gid).with('gid://gitlab/TestTwo/456', expected_type: [TestOne, TestTwo]).and_call_original
expect(described_class).to receive(:parse_gid).with('gid://gitlab/TestOne/123', { expected_type: [TestOne, TestTwo] }).and_call_original
expect(described_class).to receive(:parse_gid).with('gid://gitlab/TestTwo/456', { expected_type: [TestOne, TestTwo] }).and_call_original
expect(parse_gids.map(&:model_id)).to eq %w[123 456]
expect(parse_gids.map(&:model_class)).to eq [TestOne, TestTwo]
end

8
spec/lib/banzai/filter/wiki_link_filter_spec.rb
View File

@ -47,6 +47,14 @@ RSpec.describe Banzai::Filter::WikiLinkFilter do
expect(filtered_link.attribute('href').value).to eq(path)
end
it 'does not rewrite links to old relative wiki path' do
old_wiki_base_path = wiki.wiki_base_path.sub('/-/', '/')
path = "#{old_wiki_base_path}/#{repository_upload_folder}/a.jpg"
filtered_link = filter("<a href='#{path}'>Link</a>", wiki: wiki, page_slug: 'home').children[0]
expect(filtered_link.attribute('href').value).to eq(path)
end
end
describe "when links point to the #{Wikis::CreateAttachmentService::ATTACHMENT_PATH} folder" do

2
spec/migrations/change_public_projects_cost_factor_spec.rb
View File

@ -53,8 +53,6 @@ RSpec.describe ChangePublicProjectsCostFactor, migration: :gitlab_ci do
expect(shared_2.public_projects_minutes_cost_factor).to eq(0)
expect(shared_3.public_projects_minutes_cost_factor).to eq(1)
expect(group_1.public_projects_minutes_cost_factor).to eq(0)
schema_migrate_up!
end
end
end

38
spec/support/database/multiple_databases.rb
View File

@ -2,6 +2,15 @@
module Database
module MultipleDatabases
def run_and_cleanup(example)
# Each example may call `migrate!`, so we must ensure we are migrated down every time
schema_migrate_down!
example.run
delete_from_all_tables!(except: deletion_except_tables)
end
def skip_if_multiple_databases_not_setup
skip 'Skipping because multiple databases not set up' unless Gitlab::Database.has_config?(:ci)
end
@ -22,6 +31,21 @@ module Database
model.establish_connection(new_db_config)
end
def ensure_schema_and_empty_tables
# Ensure all schemas for both databases are migrated back
Gitlab::Database.database_base_models.each do |_, base_model|
with_reestablished_active_record_base do
reconfigure_db_connection(
model: ActiveRecord::Base,
config_model: base_model
)
schema_migrate_up!
delete_from_all_tables!(except: deletion_except_tables)
end
end
end
# The usage of this method switches temporarily used `connection_handler`
# allowing full manipulation of ActiveRecord::Base connections without
# having side effects like:
@ -109,7 +133,15 @@ RSpec.configure do |config|
end
end
config.append_after(:context, :migration) do
break if recreate_databases_and_seed_if_needed
ensure_schema_and_empty_tables
end
config.around(:each, :migration) do |example|
self.class.use_transactional_tests = false
migration_schema = example.metadata[:migration]
migration_schema = :gitlab_main if migration_schema == true
base_model = Gitlab::Database.schemas_to_base_models.fetch(migration_schema).first
@ -122,11 +154,13 @@ RSpec.configure do |config|
config_model: base_model
)
example.run
run_and_cleanup(example)
end
else
example.run
run_and_cleanup(example)
end
self.class.use_transactional_tests = true
end
end

23
spec/support/database_cleaner.rb
View File

@ -13,19 +13,6 @@ RSpec.configure do |config|
DatabaseCleaner.clean_with(:deletion)
end
config.append_after(:context, :migration) do
delete_from_all_tables!(except: ['work_item_types'])
# Postgres maximum number of columns in a table is 1600 (https://github.com/postgres/postgres/blob/de41869b64d57160f58852eab20a27f248188135/src/include/access/htup_details.h#L23-L47).
# We drop and recreate the database if any table has more than 1200 columns, just to be safe.
if any_connection_class_with_more_than_allowed_columns?
recreate_all_databases!
# Seed required data as recreating DBs will delete it
TestEnv.seed_db
end
end
config.around(:each, :delete) do |example|
self.class.use_transactional_tests = false
@ -35,14 +22,4 @@ RSpec.configure do |config|
self.class.use_transactional_tests = true
end
config.around(:each, :migration) do |example|
self.class.use_transactional_tests = false
example.run
delete_from_all_tables!(except: ['work_item_types'])
self.class.use_transactional_tests = true
end
end

13
spec/support/db_cleaner.rb
View File

@ -78,6 +78,19 @@ module DbCleaner
puts "Databases re-creation done in #{Gitlab::Metrics::System.monotonic_time - start}"
end
def recreate_databases_and_seed_if_needed
# Postgres maximum number of columns in a table is 1600 (https://github.com/postgres/postgres/blob/de41869b64d57160f58852eab20a27f248188135/src/include/access/htup_details.h#L23-L47).
# We drop and recreate the database if any table has more than 1200 columns, just to be safe.
return false unless any_connection_class_with_more_than_allowed_columns?
recreate_all_databases!
# Seed required data as recreating DBs will delete it
TestEnv.seed_db
true
end
def force_disconnect_all_connections!
cmd = <<~SQL
SELECT pg_terminate_backend(pg_stat_activity.pid)

4
spec/support/migration.rb
View File

@ -19,13 +19,9 @@ RSpec.configure do |config|
# Each example may call `migrate!`, so we must ensure we are migrated down every time
config.before(:each, :migration) do
use_fake_application_settings
schema_migrate_down!
end
config.after(:context, :migration) do
schema_migrate_up!
Gitlab::CurrentSettings.clear_in_memory_application_settings!
end
end