gitlab-org--gitlab-foss/spec/requests/api
Stan Hu d306b0d7c2 Merge branch 'use-optimistic-locking' into 'master'
Use optimistic locking

## What does this MR do?
Removes the usage of pessimistic locking in favor of optimistic which is way cheaper and doesn't block database operation.

Since this is very simple change it should be safe. If we receive `StaleObjectError` message we will reload object a retry operations in lock.

However, I still believe that we need this one: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7005 as this will reduce a load on Database and FS.
This changes a behavior from:

### Pesimistic locking (previous behavior)

#### For updating
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. enqueue: (use: transition :created -> :pending)
5. [state_machine] we are in  state created, we can go to pending
6. [state_machine] ci_pipeline.status = created
7. [state_machine] ci_pipeline.save
8. [state_machine] after_transition: (if for success): PipelineSuccessWorker on Sidekiq
9. release DB lock

#### If no update is required
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. we are in pending, we can't transition to pending, because it's forbidden
5. release DB lock

### Optimistic locking (implemented by this MR)

#### For updating
1. latest_build_status
2. enqueue: (use `transition :created -> :pending`)
3. [state_machine] we are in state created, we can go to pending
4. [state_machine] ci_pipeline.status = created
5. [state_machine] ci_pipeline.save
6. [state_machine] [save] where(lock_version: ci_pipeline.lock_version).update_all(status: :created, updated_at: Time.now)
7. [state_machine] [save] unless we_updated_row then raise ObjectInconsistentError

#### If no update is required
1. we update ci_pipeline
2. latest_build_status
3. we are in pending, we can't transition to pending, because it's forbidden

## Why was this MR needed?
We have been seeing a number of problems when we migrated Pipeline/Build processing to Sidekiq. Especially we started seeing a lot of blocking queries.

We used a pessimistic locking which doesn't seem to be required. This effectively allows us to fix our issues with blocked queries by using more efficient method of operation.

## What are the relevant issue numbers?
Issues: https://gitlab.com/gitlab-com/infrastructure/issues/623 and https://gitlab.com/gitlab-com/infrastructure/issues/584, but also there's a bunch of Merge Requests that try to improve behavior of scheduled jobs.

cc @pcarranza @yorickpeterse @stanhu

See merge request !7040
2016-10-28 14:41:24 +00:00
..
access_requests_spec.rb Invert method's naming 2016-10-03 16:57:48 +02:00
api_helpers_spec.rb API: Fix booleans not recognized as such when using the `to_boolean` helper 2016-10-27 18:21:09 +02:00
award_emoji_spec.rb Fix tests for Snippets toggling awards 2016-09-19 19:53:09 +03:00
boards_spec.rb Add support to group labels on issues board API 2016-10-19 14:58:26 -02:00
branches_spec.rb Fix branch protection API. 2016-10-24 11:33:38 +05:30
broadcast_messages_spec.rb Add BroadcastMessage API implementation 2016-08-31 17:03:18 -03:00
builds_spec.rb Fix optimistic locking 2016-10-26 11:37:23 +02:00
commit_statuses_spec.rb Grapify the commit status API 2016-10-19 12:46:27 +02:00
commits_spec.rb Added path parameter to Commits API 2016-10-24 16:43:52 +02:00
deploy_keys_spec.rb Fixes various errors when adding deploy keys caused by not exiting the control flow. 2016-10-26 12:54:47 +02:00
deployments_spec.rb Docs for API endpoints 2016-08-18 21:10:52 +02:00
doorkeeper_access_spec.rb Enable Style/EmptyLines cop, remove redundant ones 2016-07-01 21:56:17 +02:00
environments_spec.rb Expose project for environments 2016-08-18 21:10:52 +02:00
files_spec.rb Add optional 'author' param when making commits 2016-09-19 10:00:26 -07:00
fork_spec.rb Enforce the fork_project permission in Projects::CreateService 2016-09-27 13:17:56 +01:00
groups_spec.rb fix group links 404 2016-10-05 09:57:54 +01:00
internal_spec.rb Handle LFS token creation and retrieval in the same method, and in the same Redis connection. 2016-09-28 12:13:48 -05:00
issues_spec.rb Add a be_like_time matcher and use it in specs 2016-10-17 14:44:20 +01:00
keys_spec.rb adds second batch of tests changed to active tense 2016-08-09 15:11:39 +01:00
labels_spec.rb Grapify the labels API 2016-10-24 16:52:34 +02:00
lint_spec.rb Code refactor 2016-09-07 12:10:49 +02:00
members_spec.rb Fix project member access levels 2016-10-21 12:57:53 +03:00
merge_request_diffs_spec.rb Add single merge request diff API endpoint 2016-08-12 14:44:49 +03:00
merge_requests_spec.rb Allow Member.add_user to handle access requesters 2016-09-28 09:43:00 +02:00
milestones_spec.rb Allowing ">" to be used for Milestone models's title and storing the value in db as unescaped. 2016-09-29 19:28:38 -07:00
namespaces_spec.rb adds second batch of tests changed to active tense 2016-08-09 15:11:39 +01:00
notes_spec.rb Fix authored vote from notes 2016-10-24 15:24:56 +02:00
notification_settings_spec.rb Add notification_settings API calls 2016-09-09 17:08:06 +00:00
oauth_tokens_spec.rb Small refactor and syntax fixes. 2016-08-18 16:47:26 -05:00
pipelines_spec.rb Add docs on API for pipelines, plus minor fixes 2016-08-18 16:56:39 +02:00
project_hooks_spec.rb Replace undefined Grape routing code from 400 to 404 2016-10-10 16:32:32 +03:00
project_snippets_spec.rb api: expose web_url for project entities 2016-08-24 10:32:31 -04:00
projects_spec.rb Merge branch 'speed-up-api-projects-spec' into 'master' 2016-10-12 10:23:39 +00:00
repositories_spec.rb refactors tests because of gitlab-test repository changes 2016-10-11 16:33:06 +01:00
runners_spec.rb adds second batch of tests changed to active tense 2016-08-09 15:11:39 +01:00
services_spec.rb adds second batch of tests changed to active tense 2016-08-09 15:11:39 +01:00
session_spec.rb Small refactor and syntax fixes. 2016-08-18 16:47:26 -05:00
settings_spec.rb Expose the Koding application settings in the API 2016-09-29 09:12:52 -07:00
sidekiq_metrics_spec.rb Use HTTP matchers if possible 2016-06-27 20:10:42 +02:00
system_hooks_spec.rb API: Fix Sytem hooks delete behavior 2016-10-14 14:16:27 +02:00
tags_spec.rb adds second batch of tests changed to active tense 2016-08-09 15:11:39 +01:00
templates_spec.rb Create a new /templates API namespace 2016-10-12 11:09:41 +02:00
todos_spec.rb Use cache for todos counter calling TodoService 2016-08-12 17:40:03 +02:00
triggers_spec.rb Pre-create all builds for Pipeline when a trigger is received 2016-08-11 15:22:35 +02:00
users_spec.rb Make events order spec deterministic, create only 3 record instead of 5, explicitely check for events order 2016-10-24 22:56:28 +03:00
variables_spec.rb adds second batch of tests changed to active tense 2016-08-09 15:11:39 +01:00
version_spec.rb API: Version information 2016-10-12 16:47:35 +02:00