gitlab-org--gitlab-foss/spec/lib/gitlab
Stan Hu d306b0d7c2 Merge branch 'use-optimistic-locking' into 'master'
Use optimistic locking

## What does this MR do?
Removes the usage of pessimistic locking in favor of optimistic which is way cheaper and doesn't block database operation.

Since this is very simple change it should be safe. If we receive `StaleObjectError` message we will reload object a retry operations in lock.

However, I still believe that we need this one: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7005 as this will reduce a load on Database and FS.
This changes a behavior from:

### Pesimistic locking (previous behavior)

#### For updating
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. enqueue: (use: transition :created -> :pending)
5. [state_machine] we are in  state created, we can go to pending
6. [state_machine] ci_pipeline.status = created
7. [state_machine] ci_pipeline.save
8. [state_machine] after_transition: (if for success): PipelineSuccessWorker on Sidekiq
9. release DB lock

#### If no update is required
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. we are in pending, we can't transition to pending, because it's forbidden
5. release DB lock

### Optimistic locking (implemented by this MR)

#### For updating
1. latest_build_status
2. enqueue: (use `transition :created -> :pending`)
3. [state_machine] we are in state created, we can go to pending
4. [state_machine] ci_pipeline.status = created
5. [state_machine] ci_pipeline.save
6. [state_machine] [save] where(lock_version: ci_pipeline.lock_version).update_all(status: :created, updated_at: Time.now)
7. [state_machine] [save] unless we_updated_row then raise ObjectInconsistentError

#### If no update is required
1. we update ci_pipeline
2. latest_build_status
3. we are in pending, we can't transition to pending, because it's forbidden

## Why was this MR needed?
We have been seeing a number of problems when we migrated Pipeline/Build processing to Sidekiq. Especially we started seeing a lot of blocking queries.

We used a pessimistic locking which doesn't seem to be required. This effectively allows us to fix our issues with blocked queries by using more efficient method of operation.

## What are the relevant issue numbers?
Issues: https://gitlab.com/gitlab-com/infrastructure/issues/623 and https://gitlab.com/gitlab-com/infrastructure/issues/584, but also there's a bunch of Merge Requests that try to improve behavior of scheduled jobs.

cc @pcarranza @yorickpeterse @stanhu

See merge request !7040
2016-10-28 14:41:24 +00:00
..
backend Call ensure_secret_token! in secret token test's before block since it would be called in an initializer. 2016-10-06 22:40:04 -04:00
badge Fix pipeline fixtures and calls to removed method 2016-10-04 14:43:24 +02:00
bitbucket_import
checks Change the order of the access rules to check simpler first, and add specs 2016-08-12 18:27:42 -04:00
ci Merge branch '22191-delete-dynamic-envs-mr' into 'master' 2016-10-19 07:53:05 +00:00
conflict Add blob_ace_mode to conflict content response 2016-10-13 14:16:35 -05:00
data_builder refactors tests because of gitlab-test repository changes 2016-10-11 16:33:06 +01:00
database fix 2016-09-15 21:59:59 -05:00
diff Fix line commenting for the initial commit 2016-08-19 16:35:44 +01:00
downtime_check Better formatting for downtime check messages 2016-08-17 12:15:20 +02:00
email Preserve note_type and position for notes from emails 2016-10-20 16:18:53 +08:00
fogbugz_import
gfm Unfold references for group labels when moving issue to another project 2016-10-19 14:58:25 -02:00
git Enable some Rubocop cops related to new lines 2016-08-03 12:02:41 +02:00
github_import Modify GitHub importer to be retryable 2016-10-28 11:30:20 +02:00
gitlab_import Fixed gitlab.com importer missing confidential attribute 2016-09-05 17:41:48 +02:00
google_code_import Reuse LabelsFinder on Issueable#add_labels_by_names 2016-10-19 14:58:25 -02:00
graphs
import_export Fix optimistic locking 2016-10-26 11:37:23 +02:00
ldap Log LDAP lookup errors and don't swallow unrelated exceptions 2016-09-28 07:44:58 +02:00
metrics Adds response mime type to transaction metric action when it's not HTML 2016-08-25 16:33:41 +02:00
middleware Fix typo in gitlab-workhorse header 2016-08-19 12:25:52 +02:00
o_auth adds second batch of tests changed to active tense 2016-08-09 15:11:39 +01:00
saml adds second batch of tests changed to active tense 2016-08-09 15:11:39 +01:00
sanitizers
sherlock
slash_commands Fix behavior around commands with optional arguments 2016-08-18 14:29:49 -05:00
sql
template Allow Member.add_user to handle access requesters 2016-09-28 09:43:00 +02:00
asciidoc_spec.rb adds second batch of tests changed to active tense 2016-08-09 15:11:39 +01:00
auth_spec.rb Handle LFS token creation and retrieval in the same method, and in the same Redis connection. 2016-09-28 12:13:48 -05:00
award_emoji_spec.rb
blame_spec.rb
changes_list_spec.rb api for generating new merge request 2016-08-11 23:37:00 +07:00
closing_issue_extractor_spec.rb Don’t close issues on original project from a fork 2016-08-04 12:38:08 +02:00
color_schemes_spec.rb
current_settings_spec.rb
database_spec.rb
downtime_check_spec.rb
exclusive_lease_spec.rb Don't schedule ProjectCacheWorker unless needed 2016-10-25 16:02:36 +02:00
git_access_spec.rb Backport git access spec changes from EE 2016-10-18 17:28:57 +01:00
git_access_wiki_spec.rb Fix specs after renaming authentication_capabilities 2016-09-16 11:12:22 +02:00
git_spec.rb Add spec covering 'committer_hash' 2016-09-20 10:15:43 -07:00
highlight_spec.rb Fix specs 2016-08-01 13:24:06 +02:00
identifier_spec.rb Refactor Gitlab::Identifier 2016-10-05 12:40:17 +02:00
incoming_email_spec.rb
key_fingerprint_spec.rb
lazy_spec.rb
lfs_token_spec.rb Handle LFS token creation and retrieval in the same method, and in the same Redis connection. 2016-09-28 12:13:48 -05:00
markup_helper_spec.rb
metrics_spec.rb Tracking of custom events 2016-08-17 10:04:04 +02:00
optimistic_locking_spec.rb Make retry_lock to not be infinite 2016-10-27 12:34:35 +01:00
popen_spec.rb Use 'git update-ref' for safer web commits 2016-09-02 11:49:40 +02:00
project_search_results_spec.rb adds second batch of tests changed to active tense 2016-08-09 15:11:39 +01:00
redis_spec.rb Updated mail_room and added sentinel support to Reply by Email 2016-10-26 05:02:47 +02:00
reference_extractor_spec.rb
regex_spec.rb
search_results_spec.rb Clean up search result classes 2016-09-06 10:12:55 +03:00
snippet_search_results_spec.rb Clean up search result classes 2016-09-06 10:12:55 +03:00
themes_spec.rb
upgrader_spec.rb adds second batch of tests changed to active tense 2016-08-09 15:11:39 +01:00
uploads_transfer_spec.rb
url_builder_spec.rb
url_sanitizer_spec.rb
user_access_spec.rb changes default_branch_protection to allow devs_can_merge protection option aswell 2016-08-04 21:15:26 +01:00
version_info_spec.rb
workhorse_spec.rb Use base SHA for patches and diffs 2016-09-22 18:49:31 +01:00