add initial smoke tests and documentation
This commit is contained in:
parent
c5bf09e7a4
commit
d67936b68f
|
@ -0,0 +1,16 @@
|
|||
# Smoke Tests
|
||||
|
||||
It is imperative in any testing suite that we have Smoke Tests. In short, smoke tests are will run quick sanity
|
||||
end-to-end functional tests from GitLab QA and are designed to run against the specified environment to ensure that
|
||||
basic functionality is working.
|
||||
|
||||
Currently, our suite consists of this basic functionality coverage:
|
||||
|
||||
- User Login (Standard Auth)
|
||||
- Project Creation
|
||||
- Issue Creation
|
||||
- Merge Request Creation
|
||||
|
||||
---
|
||||
|
||||
[Return to Testing documentation](index.md)
|
|
@ -120,6 +120,14 @@ running feature tests (i.e. using Capybara) against it.
|
|||
The actual test scenarios and steps are [part of GitLab Rails] so that they're
|
||||
always in-sync with the codebase.
|
||||
|
||||
### Smoke tests
|
||||
|
||||
Smoke tests are quick tests that may be run at any time (especially after the pre-deployment migrations).
|
||||
|
||||
Much like feature tests - these tests run against the UI and ensure that basic functionality is working.
|
||||
|
||||
> See [Smoke Tests](smoke.md) for more information.
|
||||
|
||||
Read a separate document about [end-to-end tests](end_to_end_tests.md) to
|
||||
learn more.
|
||||
|
||||
|
|
12
qa/README.md
12
qa/README.md
|
@ -35,7 +35,7 @@ following call would login to a local [GDK] instance and run all specs in
|
|||
`qa/specs/features`:
|
||||
|
||||
```
|
||||
bin/qa Test::Instance http://localhost:3000
|
||||
bin/qa Test::Instance::All http://localhost:3000
|
||||
```
|
||||
|
||||
### Writing tests
|
||||
|
@ -48,14 +48,14 @@ You can also supply specific tests to run as another parameter. For example, to
|
|||
run the repository-related specs, you can execute:
|
||||
|
||||
```
|
||||
bin/qa Test::Instance http://localhost qa/specs/features/repository/
|
||||
bin/qa Test::Instance::All http://localhost qa/specs/features/repository/
|
||||
```
|
||||
|
||||
Since the arguments would be passed to `rspec`, you could use all `rspec`
|
||||
options there. For example, passing `--backtrace` and also line number:
|
||||
|
||||
```
|
||||
bin/qa Test::Instance http://localhost qa/specs/features/project/create_spec.rb:3 --backtrace
|
||||
bin/qa Test::Instance::All http://localhost qa/specs/features/project/create_spec.rb:3 --backtrace
|
||||
```
|
||||
|
||||
### Overriding the authenticated user
|
||||
|
@ -67,7 +67,7 @@ If you need to authenticate as a different user, you can provide the
|
|||
`GITLAB_USERNAME` and `GITLAB_PASSWORD` environment variables:
|
||||
|
||||
```
|
||||
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bin/qa Test::Instance https://gitlab.example.com
|
||||
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password bin/qa Test::Instance::All https://gitlab.example.com
|
||||
```
|
||||
|
||||
If your user doesn't have permission to default sandbox group
|
||||
|
@ -75,13 +75,13 @@ If your user doesn't have permission to default sandbox group
|
|||
`GITLAB_SANDBOX_NAME`:
|
||||
|
||||
```
|
||||
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance https://gitlab.example.com
|
||||
GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance::All https://gitlab.example.com
|
||||
```
|
||||
|
||||
In addition, the `GITLAB_USER_TYPE` can be set to "ldap" to sign in as an LDAP user:
|
||||
|
||||
```
|
||||
GITLAB_USER_TYPE=ldap GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance https://gitlab.example.com
|
||||
GITLAB_USER_TYPE=ldap GITLAB_USERNAME=jsmith GITLAB_PASSWORD=password GITLAB_SANDBOX_NAME=jsmith-qa-sandbox bin/qa Test::Instance::All https://gitlab.example.com
|
||||
```
|
||||
|
||||
All [supported environment variables are here](https://gitlab.com/gitlab-org/gitlab-qa#supported-environment-variables).
|
||||
|
|
6
qa/qa.rb
6
qa/qa.rb
|
@ -77,14 +77,16 @@ module QA
|
|||
#
|
||||
autoload :Bootable, 'qa/scenario/bootable'
|
||||
autoload :Actable, 'qa/scenario/actable'
|
||||
autoload :Taggable, 'qa/scenario/taggable'
|
||||
autoload :Template, 'qa/scenario/template'
|
||||
|
||||
##
|
||||
# Test scenario entrypoints.
|
||||
#
|
||||
module Test
|
||||
autoload :Instance, 'qa/scenario/test/instance'
|
||||
module Instance
|
||||
autoload :All, 'qa/scenario/test/instance/all'
|
||||
autoload :Smoke, 'qa/scenario/test/instance/smoke'
|
||||
end
|
||||
|
||||
module Integration
|
||||
autoload :Github, 'qa/scenario/test/integration/github'
|
||||
|
|
|
@ -1,17 +0,0 @@
|
|||
module QA
|
||||
module Scenario
|
||||
module Taggable
|
||||
# rubocop:disable Gitlab/ModuleWithInstanceVariables
|
||||
|
||||
def tags(*tags)
|
||||
@tags = tags
|
||||
end
|
||||
|
||||
def focus
|
||||
@tags.to_a
|
||||
end
|
||||
|
||||
# rubocop:enable Gitlab/ModuleWithInstanceVariables
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,15 +1,36 @@
|
|||
module QA
|
||||
module Scenario
|
||||
class Template
|
||||
def self.perform(*args)
|
||||
new.tap do |scenario|
|
||||
yield scenario if block_given?
|
||||
break scenario.perform(*args)
|
||||
class << self
|
||||
def perform(*args)
|
||||
new.tap do |scenario|
|
||||
yield scenario if block_given?
|
||||
break scenario.perform(*args)
|
||||
end
|
||||
end
|
||||
|
||||
def tags(*tags)
|
||||
@tags = tags
|
||||
end
|
||||
|
||||
def focus
|
||||
@tags.to_a
|
||||
end
|
||||
end
|
||||
|
||||
def perform(*_args)
|
||||
raise NotImplementedError
|
||||
def perform(address, *rspec_options)
|
||||
Runtime::Scenario.define(:gitlab_address, address)
|
||||
|
||||
Specs::Runner.perform do |specs|
|
||||
specs.tty = true
|
||||
specs.tags = self.class.focus
|
||||
specs.options =
|
||||
if rspec_options.any?
|
||||
rspec_options
|
||||
else
|
||||
File.expand_path('../../specs/features', __dir__)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
|
@ -1,36 +0,0 @@
|
|||
module QA
|
||||
module Scenario
|
||||
module Test
|
||||
##
|
||||
# Base class for running the suite against any GitLab instance,
|
||||
# including staging and on-premises installation.
|
||||
#
|
||||
class Instance < Template
|
||||
include Bootable
|
||||
extend Taggable
|
||||
|
||||
tags :core
|
||||
|
||||
def perform(address, *rspec_options)
|
||||
Runtime::Scenario.define(:gitlab_address, address)
|
||||
|
||||
##
|
||||
# Perform before hooks, which are different for CE and EE
|
||||
#
|
||||
Runtime::Release.perform_before_hooks
|
||||
|
||||
Specs::Runner.perform do |specs|
|
||||
specs.tty = true
|
||||
specs.tags = self.class.focus
|
||||
specs.options =
|
||||
if rspec_options.any?
|
||||
rspec_options
|
||||
else
|
||||
::File.expand_path('../../specs/features', __dir__)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
|
@ -0,0 +1,15 @@
|
|||
module QA
|
||||
module Scenario
|
||||
module Test
|
||||
##
|
||||
# Base class for running the suite against any GitLab instance,
|
||||
# including staging and on-premises installation.
|
||||
#
|
||||
module Instance
|
||||
class All < Template
|
||||
include Bootable
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
|
@ -0,0 +1,17 @@
|
|||
module QA
|
||||
module Scenario
|
||||
module Test
|
||||
module Instance
|
||||
##
|
||||
# Base class for running the suite against any GitLab instance,
|
||||
# including staging and on-premises installation.
|
||||
#
|
||||
class Smoke < Template
|
||||
include Bootable
|
||||
|
||||
tags :smoke
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,7 +1,7 @@
|
|||
require 'securerandom'
|
||||
|
||||
module QA
|
||||
describe 'API basics', :core do
|
||||
describe 'API basics' do
|
||||
before(:context) do
|
||||
@api_client = Runtime::API::Client.new(:gitlab)
|
||||
end
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'API users', :core do
|
||||
describe 'API users' do
|
||||
before(:context) do
|
||||
@api_client = Runtime::API::Client.new(:gitlab)
|
||||
end
|
||||
|
|
|
@ -0,0 +1,15 @@
|
|||
module QA
|
||||
describe 'basic user login', :smoke do
|
||||
it 'user logs in using basic credentials' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
||||
# TODO, since `Signed in successfully` message was removed
|
||||
# this is the only way to tell if user is signed in correctly.
|
||||
#
|
||||
Page::Menu::Main.perform do |menu|
|
||||
expect(menu).to have_personal_area
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'LDAP user login', :ldap do
|
||||
describe 'LDAP user login', :orchestrated, :ldap do
|
||||
before do
|
||||
Runtime::Env.user_type = 'ldap'
|
||||
end
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'create a new group', :mattermost do
|
||||
describe 'create a new group', :orchestrated, :mattermost do
|
||||
it 'creating a group with a mattermost team' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'logging in to Mattermost', :mattermost do
|
||||
describe 'logging in to Mattermost', :orchestrated, :mattermost do
|
||||
it 'can use gitlab oauth' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login) do
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'creates a merge request', :core do
|
||||
describe 'creates a merge request with milestone' do
|
||||
it 'user creates a new merge request' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
@ -29,4 +29,25 @@ module QA
|
|||
end
|
||||
end
|
||||
end
|
||||
|
||||
describe 'creates a merge request', :smoke do
|
||||
it 'user creates a new merge request' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
||||
current_project = Factory::Resource::Project.fabricate! do |project|
|
||||
project.name = 'project-with-merge-request'
|
||||
end
|
||||
|
||||
Factory::Resource::MergeRequest.fabricate! do |merge_request|
|
||||
merge_request.title = 'This is a merge request'
|
||||
merge_request.description = 'Great feature'
|
||||
merge_request.project = current_project
|
||||
end
|
||||
|
||||
expect(page).to have_content('This is a merge request')
|
||||
expect(page).to have_content('Great feature')
|
||||
expect(page).to have_content(/Opened [\w\s]+ ago/)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'merge request rebase', :core do
|
||||
describe 'merge request rebase' do
|
||||
it 'rebases source branch of merge request' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'merge request squash commits', :core do
|
||||
describe 'merge request squash commits' do
|
||||
it 'when squash commits is marked before merge' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'activity page', :core do
|
||||
describe 'activity page' do
|
||||
it 'push creates an event in the activity page' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'deploy keys support', :core do
|
||||
describe 'deploy keys support' do
|
||||
it 'user adds a deploy key' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'secret variables support', :core do
|
||||
describe 'secret variables support' do
|
||||
it 'user adds a secret variable' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
|
|
@ -1,7 +1,7 @@
|
|||
require 'pathname'
|
||||
|
||||
module QA
|
||||
describe 'Auto Devops', :kubernetes do
|
||||
describe 'Auto Devops', :orchestrated, :kubernetes do
|
||||
after do
|
||||
@cluster&.remove!
|
||||
end
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'creates issue', :core do
|
||||
describe 'creates issue', :smoke do
|
||||
let(:issue_title) { 'issue title' }
|
||||
|
||||
it 'user creates issue' do
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'create a new project', :core do
|
||||
describe 'create a new project', :smoke do
|
||||
it 'user creates a new project' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
|
|
@ -1,7 +1,7 @@
|
|||
require 'digest/sha1'
|
||||
|
||||
module QA
|
||||
describe 'cloning code using a deploy key', :core, :docker do
|
||||
describe 'cloning code using a deploy key', :docker do
|
||||
def login
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'Project fork', :core do
|
||||
describe 'Project fork' do
|
||||
it 'can submit merge requests to upstream master' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'user imports a GitHub repo', :core, :github do
|
||||
describe 'user imports a GitHub repo', :orchestrated, :github do
|
||||
let(:imported_project) do
|
||||
Factory::Resource::ProjectImportedFromGithub.fabricate! do |project|
|
||||
project.name = 'imported-project'
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'CI/CD Pipelines', :core, :docker do
|
||||
describe 'CI/CD Pipelines', :orchestrated, :docker do
|
||||
let(:executor) { "qa-runner-#{Time.now.to_i}" }
|
||||
|
||||
after do
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'Wiki Functionality', :core do
|
||||
describe 'Wiki Functionality' do
|
||||
def login
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
Page::Main::Login.act { sign_in_using_credentials }
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'clone code from the repository', :core do
|
||||
describe 'clone code from the repository' do
|
||||
context 'with regular account over http' do
|
||||
let(:location) do
|
||||
Page::Project::Show.act do
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'branch protection support', :core do
|
||||
describe 'branch protection support' do
|
||||
let(:branch_name) { 'protected-branch' }
|
||||
let(:commit_message) { 'Protected push commit message' }
|
||||
let(:project) do
|
||||
|
|
|
@ -1,5 +1,5 @@
|
|||
module QA
|
||||
describe 'push code to repository', :core do
|
||||
describe 'push code to repository' do
|
||||
context 'with regular account over http' do
|
||||
it 'user pushes code to the repository' do
|
||||
Runtime::Browser.visit(:gitlab, Page::Main::Login)
|
||||
|
|
|
@ -14,7 +14,13 @@ module QA
|
|||
def perform
|
||||
args = []
|
||||
args.push('--tty') if tty
|
||||
tags.to_a.each { |tag| args.push(['-t', tag.to_s]) }
|
||||
|
||||
if tags.any?
|
||||
tags.each { |tag| args.push(['-t', tag.to_s]) }
|
||||
else
|
||||
args.push(%w[-t ~orchestrated])
|
||||
end
|
||||
|
||||
args.push(options)
|
||||
|
||||
Runtime::Browser.configure!
|
||||
|
|
|
@ -1,10 +1,4 @@
|
|||
describe QA::Scenario::Test::Instance do
|
||||
subject do
|
||||
Class.new(described_class) do
|
||||
tags :rspec
|
||||
end
|
||||
end
|
||||
|
||||
describe QA::Scenario::Test::Instance::All do
|
||||
context '#perform' do
|
||||
let(:arguments) { spy('Runtime::Scenario') }
|
||||
let(:release) { spy('Runtime::Release') }
|
||||
|
@ -26,16 +20,16 @@ describe QA::Scenario::Test::Instance do
|
|||
end
|
||||
|
||||
context 'no paths' do
|
||||
it 'should call runner with default arguments' do
|
||||
it 'calls runner with default arguments' do
|
||||
subject.perform("test")
|
||||
|
||||
expect(runner).to have_received(:options=)
|
||||
.with(::File.expand_path('../../../qa/specs/features', __dir__))
|
||||
.with(::File.expand_path('../../../../../qa/specs/features', __dir__))
|
||||
end
|
||||
end
|
||||
|
||||
context 'specifying paths' do
|
||||
it 'should call runner with paths' do
|
||||
it 'calls runner with paths' do
|
||||
subject.perform('test', 'path1', 'path2')
|
||||
|
||||
expect(runner).to have_received(:options=).with(%w[path1 path2])
|
|
@ -0,0 +1,45 @@
|
|||
describe QA::Scenario::Test::Instance::Smoke do
|
||||
subject { Class.new(described_class) { tags :smoke } }
|
||||
|
||||
context '#perform' do
|
||||
let(:arguments) { spy('Runtime::Scenario') }
|
||||
let(:release) { spy('Runtime::Release') }
|
||||
let(:runner) { spy('Specs::Runner') }
|
||||
|
||||
before do
|
||||
stub_const('QA::Runtime::Release', release)
|
||||
stub_const('QA::Runtime::Scenario', arguments)
|
||||
stub_const('QA::Specs::Runner', runner)
|
||||
|
||||
allow(runner).to receive(:perform).and_yield(runner)
|
||||
end
|
||||
|
||||
it 'sets an address of the subject' do
|
||||
subject.perform("hello")
|
||||
|
||||
expect(arguments).to have_received(:define)
|
||||
.with(:gitlab_address, "hello")
|
||||
end
|
||||
|
||||
it 'has a smoke tag' do
|
||||
expect(subject.focus).to eq([:smoke]) # rubocop:disable Focus
|
||||
end
|
||||
|
||||
context 'no paths' do
|
||||
it 'calls runner with default arguments' do
|
||||
subject.perform("test")
|
||||
|
||||
expect(runner).to have_received(:options=)
|
||||
.with(File.expand_path('../../../../../qa/specs/features', __dir__))
|
||||
end
|
||||
end
|
||||
|
||||
context 'specifying paths' do
|
||||
it 'calls runner with paths' do
|
||||
subject.perform('test', 'path1', 'path2')
|
||||
|
||||
expect(runner).to have_received(:options=).with(%w[path1 path2])
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
Loading…
Reference in New Issue