gitlab-org--gitlab-foss/app/models
Stan Hu d306b0d7c2 Merge branch 'use-optimistic-locking' into 'master'
Use optimistic locking

## What does this MR do?
Removes the usage of pessimistic locking in favor of optimistic which is way cheaper and doesn't block database operation.

Since this is very simple change it should be safe. If we receive `StaleObjectError` message we will reload object a retry operations in lock.

However, I still believe that we need this one: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7005 as this will reduce a load on Database and FS.
This changes a behavior from:

### Pesimistic locking (previous behavior)

#### For updating
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. enqueue: (use: transition :created -> :pending)
5. [state_machine] we are in  state created, we can go to pending
6. [state_machine] ci_pipeline.status = created
7. [state_machine] ci_pipeline.save
8. [state_machine] after_transition: (if for success): PipelineSuccessWorker on Sidekiq
9. release DB lock

#### If no update is required
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. we are in pending, we can't transition to pending, because it's forbidden
5. release DB lock

### Optimistic locking (implemented by this MR)

#### For updating
1. latest_build_status
2. enqueue: (use `transition :created -> :pending`)
3. [state_machine] we are in state created, we can go to pending
4. [state_machine] ci_pipeline.status = created
5. [state_machine] ci_pipeline.save
6. [state_machine] [save] where(lock_version: ci_pipeline.lock_version).update_all(status: :created, updated_at: Time.now)
7. [state_machine] [save] unless we_updated_row then raise ObjectInconsistentError

#### If no update is required
1. we update ci_pipeline
2. latest_build_status
3. we are in pending, we can't transition to pending, because it's forbidden

## Why was this MR needed?
We have been seeing a number of problems when we migrated Pipeline/Build processing to Sidekiq. Especially we started seeing a lot of blocking queries.

We used a pessimistic locking which doesn't seem to be required. This effectively allows us to fix our issues with blocked queries by using more efficient method of operation.

## What are the relevant issue numbers?
Issues: https://gitlab.com/gitlab-com/infrastructure/issues/623 and https://gitlab.com/gitlab-com/infrastructure/issues/584, but also there's a bunch of Merge Requests that try to improve behavior of scheduled jobs.

cc @pcarranza @yorickpeterse @stanhu

See merge request !7040
2016-10-28 14:41:24 +00:00
..
ci Add tests for optimistic locking 2016-10-26 11:37:23 +02:00
concerns Fix and improve Sortable.highest_label_priority 2016-10-27 21:26:56 -03:00
cycle_analytics Fix the "Commits" section of the cycle analytics summary. 2016-09-26 18:38:40 +05:30
hooks Scope hooks thal will run for confidential issues 2016-08-31 18:34:45 -03:00
issue Implement (some) comments from @DouweM's review. 2016-09-20 18:13:11 +05:30
members Remove redundant class_name and foreign_key overrides 2016-10-24 22:29:48 +02:00
merge_request Implement (some) comments from @DouweM's review. 2016-09-20 18:13:11 +05:30
network
project_services Finish updates to use JIRA gem 2016-10-26 15:02:16 -02:00
protected_branch Backport changes from gitlab-org/gitlab-ee!581 to CE. 2016-08-16 11:05:14 +05:30
.gitkeep
ability.rb factor out a RuleSet so that delegate! retains @cannot 2016-08-30 11:39:22 -07:00
abuse_report.rb Add markdown cache columns to the database, but don't use them yet 2016-10-07 02:54:25 +01:00
appearance.rb Add markdown cache columns to the database, but don't use them yet 2016-10-07 02:54:25 +01:00
application_setting.rb Add markdown cache columns to the database, but don't use them yet 2016-10-07 02:54:25 +01:00
audit_event.rb
award_emoji.rb
blob.rb Handle encoding in non-binary Blob instances 2016-09-14 14:15:31 +02:00
board.rb Improve project_with_board factory to create the default lists 2016-09-26 14:42:38 -03:00
broadcast_message.rb Add markdown cache columns to the database, but don't use them yet 2016-10-07 02:54:25 +01:00
commit.rb Fix Commit#status, feedback: 2016-09-14 01:25:26 +08:00
commit_range.rb Enable Lint/StringConversionInInterpolation cop and autocorrect offenses 2016-10-03 16:09:57 +02:00
commit_status.rb Fix optimistic locking 2016-10-26 11:37:23 +02:00
compare.rb Refactor straight compare diff code 2016-10-12 16:32:57 +03:00
cycle_analytics.rb updated var name based on feedback 2016-10-13 09:19:30 +02:00
deploy_key.rb Refactor LFS token logic to use a Redis key instead of a DB field, making it a 1 use only token. 2016-09-15 12:21:00 -05:00
deploy_keys_project.rb
deployment.rb Only create refs for new deployments 2016-10-20 14:21:40 +02:00
diff_note.rb Optimize discussion notes resolving and unresolving 2016-09-06 12:14:09 -03:00
discussion.rb Optimize discussion notes resolving and unresolving 2016-09-06 12:14:09 -03:00
email.rb refactor(email): use setter method instead AR callbacks 2016-10-23 17:30:07 +05:00
environment.rb Use deployment IID when saving refs 2016-10-20 14:17:22 +02:00
event.rb Differentiate the expire from leave event 2016-10-20 00:26:45 +00:00
external_issue.rb Prevent wrong markdown on issue ids when project has Jira service activated 2016-10-19 13:46:04 -02:00
forked_project_link.rb
generic_commit_status.rb
global_label.rb Add markdown cache columns to the database, but don't use them yet 2016-10-07 02:54:25 +01:00
global_milestone.rb Add markdown cache columns to the database, but don't use them yet 2016-10-07 02:54:25 +01:00
group.rb Revert "Revert "Change "Group#web_url" to return "/groups/twitter" rather than "/twitter"."" 2016-10-27 16:16:15 +03:00
group_label.rb Add subject to group and projects labels which return group/project 2016-10-19 14:58:27 -02:00
identity.rb
issue.rb Merge branch 'master' into issue-board-sidebar 2016-10-26 08:47:09 +01:00
key.rb Add simple identifier to public SSH keys 2016-08-02 06:56:23 +01:00
label.rb Use join instead of subquery on Label.unprioritized scope 2016-10-19 14:58:27 -02:00
label_link.rb squashed - fixed label and milestone association problems, updated specs and refactored reader class a bit 2016-08-01 09:57:40 +02:00
label_priority.rb Add LabelPriority model 2016-10-19 14:58:27 -02:00
legacy_diff_note.rb Move #to_discussion to NoteOnDiff 2016-08-30 16:30:42 +01:00
lfs_object.rb
lfs_objects_project.rb
list.rb Fix issue board related controllers to expose label priority per project 2016-10-19 14:58:27 -02:00
member.rb Make Member#add_user set access_level for requesters 2016-10-03 11:24:16 +02:00
merge_request.rb Remove redundant class_name and foreign_key overrides 2016-10-24 22:29:48 +02:00
merge_request_diff.rb Also keep commits from source_project around, feedback: 2016-10-25 00:08:30 +08:00
merge_requests_closing_issues.rb Implement a second round of review comments from @DouweM. 2016-09-21 00:47:37 +05:30
milestone.rb Add markdown cache columns to the database, but don't use them yet 2016-10-07 02:54:25 +01:00
namespace.rb Correct namespace validation to forbid bad names #21077 2016-10-07 13:46:59 -05:00
note.rb Use CacheMarkdownField for notes 2016-10-07 02:54:26 +01:00
notification_setting.rb Split notification integration into another branch 2016-09-14 19:34:24 +08:00
oauth_access_token.rb
personal_access_token.rb
personal_snippet.rb
project.rb Remove redundant class_name and foreign_key overrides 2016-10-24 22:29:48 +02:00
project_feature.rb Add visibility level to project repository 2016-10-17 18:12:18 -02:00
project_group_link.rb Merge branch 'api-fix-project-group-sharing' into 'security' 2016-10-11 20:36:26 +02:00
project_import_data.rb
project_label.rb Fix max number of permitted priorities per project label 2016-10-19 14:58:28 -02:00
project_snippet.rb
project_team.rb Allow Member.add_user to handle access requesters 2016-09-28 09:43:00 +02:00
project_wiki.rb Add method missing from EE 2016-08-12 21:56:40 -04:00
protected_branch.rb Backport changes from gitlab-org/gitlab-ee!581 to CE. 2016-08-16 11:05:14 +05:30
release.rb Add markdown cache columns to the database, but don't use them yet 2016-10-07 02:54:25 +01:00
repository.rb Remove duplicate code in repository cache clearing 2016-10-23 10:45:08 -07:00
security_event.rb
sent_notification.rb
service.rb Merge remote-tracking branch 'upstream/master' into pipeline-emails 2016-10-08 07:51:07 +00:00
snippet.rb Add markdown cache columns to the database, but don't use them yet 2016-10-07 02:54:25 +01:00
spam_log.rb Allow SpamLog to be submitted as ham 2016-08-15 13:18:15 -05:00
subscription.rb
todo.rb Fix and improve Sortable.highest_label_priority 2016-10-27 21:26:56 -03:00
tree.rb
trending_project.rb Precalculate trending projects 2016-10-10 12:27:08 +02:00
u2f_registration.rb Allow naming U2F devices. 2016-08-18 22:12:02 +05:30
user.rb Merge branch 'master' into 5905-duplicate-email-errors 2016-10-27 16:07:56 -04:00
user_agent_detail.rb Further refactor and syntax fixes. 2016-08-15 17:20:57 -05:00
users_star_project.rb
wiki_page.rb WikiPage should have a slug even when not persisted. 2016-07-25 09:22:47 +05:30