gitlab-org--gitlab-foss/db/fixtures/development/17_cycle_analytics.rb

235 lines
7.1 KiB
Ruby
Raw Normal View History

Fix race conditions for AuthorizedProjectsWorker There were two cases that could be problematic: 1. Because sometimes AuthorizedProjectsWorker would be scheduled in a transaction it was possible for a job to run/complete before a COMMIT; resulting in it either producing an error, or producing no new data. 2. When scheduling jobs the code would not wait until completion. This could lead to a user creating a project and then immediately trying to push to it. Usually this will work fine, but given enough load it might take a few seconds before a user has access. The first one is problematic, the second one is mostly just annoying (but annoying enough to warrant a solution). This commit changes two things to deal with this: 1. Sidekiq scheduling now takes places after a COMMIT, this is ensured by scheduling using Rails' after_commit hook instead of doing so in an arbitrary method. 2. When scheduling jobs the calling thread now waits for all jobs to complete. Solution 2 requires tracking of job completions. Sidekiq provides a way to find a job by its ID, but this involves scanning over the entire queue; something that is very in-efficient for large queues. As such a more efficient solution is necessary. There are two main Gems that can do this in a more efficient manner: * sidekiq-status * sidekiq_status No, this is not a joke. Both Gems do a similar thing (but slightly different), and the only difference in their name is a dash vs an underscore. Both Gems however provide far more than just checking if a job has been completed, and both have their problems. sidekiq-status does not appear to be actively maintained, with the last release being in 2015. It also has some issues during testing as API calls are not stubbed in any way. sidekiq_status on the other hand does not appear to be very popular, and introduces a similar amount of code. Because of this I opted to write a simple home grown solution. After all, all we need is storing a job ID somewhere so we can efficiently look it up; we don't need extra web UIs (as provided by sidekiq-status) or complex APIs to update progress, etc. This is where Gitlab::SidekiqStatus comes in handy. This namespace contains some code used for tracking, removing, and looking up job IDs; all without having to scan over an entire queue. Data is removed explicitly, but also expires automatically just in case. Using this API we can now schedule jobs in a fork-join like manner: we schedule the jobs in Sidekiq, process them in parallel, then wait for completion. By using Sidekiq we can leverage all the benefits such as being able to scale across multiple cores and hosts, retrying failed jobs, etc. The one downside is that we need to make sure we can deal with unexpected increases in job processing timings. To deal with this the class Gitlab::JobWaiter (used for waiting for jobs to complete) will only wait a number of seconds (30 by default). Once this timeout is reached it will simply return. For GitLab.com almost all AuthorizedProjectWorker jobs complete in seconds, only very rarely do we spike to job timings of around a minute. These in turn seem to be the result of external factors (e.g. deploys), in which case a user is most likely not able to use the system anyway. In short, this new solution should ensure that jobs are processed properly and that in almost all cases a user has access to their resources whenever they need to have access.
2017-01-22 12:22:02 -05:00
require './spec/support/sidekiq'
require './spec/support/helpers/test_env'
class Gitlab::Seeder::CycleAnalytics
2016-09-14 10:10:31 -04:00
def initialize(project, perf: false)
@project = project
@user = User.admins.first
2016-09-14 10:10:31 -04:00
@issue_count = perf ? 1000 : 5
end
def seed_metrics!
@issue_count.times do |index|
# Issue
Timecop.travel 5.days.from_now
title = "#{FFaker::Product.brand}-#{FFaker::Product.brand}-#{rand(1000)}"
issue = Issue.create(project: @project, title: title, author: @user)
issue_metrics = issue.metrics
# Milestones / Labels
Timecop.travel 5.days.from_now
if index.even?
issue_metrics.first_associated_with_milestone_at = rand(6..12).hours.from_now
else
issue_metrics.first_added_to_board_at = rand(6..12).hours.from_now
end
# Commit
Timecop.travel 5.days.from_now
issue_metrics.first_mentioned_in_commit_at = rand(6..12).hours.from_now
# MR
Timecop.travel 5.days.from_now
branch_name = "#{FFaker::Product.brand}-#{FFaker::Product.brand}-#{rand(1000)}"
@project.repository.add_branch(@user, branch_name, 'master')
merge_request = MergeRequest.create(target_project: @project, source_project: @project, source_branch: branch_name, target_branch: 'master', title: branch_name, author: @user)
merge_request_metrics = merge_request.metrics
# MR closing issues
Timecop.travel 5.days.from_now
MergeRequestsClosingIssues.create!(issue: issue, merge_request: merge_request)
# Merge
Timecop.travel 5.days.from_now
merge_request_metrics.merged_at = rand(6..12).hours.from_now
# Start build
Timecop.travel 5.days.from_now
merge_request_metrics.latest_build_started_at = rand(6..12).hours.from_now
# Finish build
Timecop.travel 5.days.from_now
merge_request_metrics.latest_build_finished_at = rand(6..12).hours.from_now
# Deploy to production
Timecop.travel 5.days.from_now
merge_request_metrics.first_deployed_to_production_at = rand(6..12).hours.from_now
issue_metrics.save!
merge_request_metrics.save!
print '.'
end
end
def seed!
Sidekiq::Worker.skipping_transaction_check do
Sidekiq::Testing.inline! do
issues = create_issues
puts '.'
# Stage 1
Timecop.travel 5.days.from_now
add_milestones_and_list_labels(issues)
print '.'
# Stage 2
Timecop.travel 5.days.from_now
branches = mention_in_commits(issues)
print '.'
# Stage 3
Timecop.travel 5.days.from_now
merge_requests = create_merge_requests_closing_issues(issues, branches)
print '.'
# Stage 4
Timecop.travel 5.days.from_now
run_builds(merge_requests)
print '.'
# Stage 5
Timecop.travel 5.days.from_now
merge_merge_requests(merge_requests)
print '.'
# Stage 6 / 7
Timecop.travel 5.days.from_now
deploy_to_production(merge_requests)
print '.'
end
end
print '.'
end
private
def create_issues
2016-09-14 10:10:31 -04:00
Array.new(@issue_count) do
issue_params = {
title: "Cycle Analytics: #{FFaker::Lorem.sentence(6)}",
description: FFaker::Lorem.sentence,
state: 'opened',
assignees: [@project.team.users.sample]
}
Issues::CreateService.new(@project, @project.team.users.sample, issue_params).execute
end
end
def add_milestones_and_list_labels(issues)
issues.shuffle.map.with_index do |issue, index|
Timecop.travel 12.hours.from_now
if index.even?
issue.update(milestone: @project.milestones.sample)
else
label_name = "#{FFaker::Product.brand}-#{FFaker::Product.brand}-#{rand(1000)}"
list_label = FactoryBot.create(:label, title: label_name, project: issue.project)
FactoryBot.create(:list, board: FactoryBot.create(:board, project: issue.project), label: list_label)
issue.update(labels: [list_label])
end
issue
end
end
def mention_in_commits(issues)
issues.map do |issue|
Timecop.travel 12.hours.from_now
branch_name = filename = "#{FFaker::Product.brand}-#{FFaker::Product.brand}-#{rand(1000)}"
issue.project.repository.add_branch(@user, branch_name, 'master')
commit_sha = issue.project.repository.create_file(@user, filename, "content", message: "Commit for #{issue.to_reference}", branch_name: branch_name)
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
issue.project.repository.commit(commit_sha)
GitPushService.new(issue.project,
@user,
oldrev: issue.project.repository.commit("master").sha,
newrev: commit_sha,
ref: 'refs/heads/master').execute
branch_name
end
end
def create_merge_requests_closing_issues(issues, branches)
issues.zip(branches).map do |issue, branch|
Timecop.travel 12.hours.from_now
opts = {
title: 'Cycle Analytics merge_request',
description: "Fixes #{issue.to_reference}",
source_branch: branch,
target_branch: 'master'
}
MergeRequests::CreateService.new(issue.project, @user, opts).execute
end
end
def run_builds(merge_requests)
merge_requests.each do |merge_request|
Timecop.travel 12.hours.from_now
service = Ci::CreatePipelineService.new(merge_request.project,
@user,
ref: "refs/heads/#{merge_request.source_branch}")
2017-05-24 09:13:51 -04:00
pipeline = service.execute(:push, ignore_skip_ci: true, save_on_errors: false)
Squashed commit of the following: commit 931d6ab0e025b0268d94e455f736b09a025e0578 Merge: b34d165320d 93846eb152f Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Nov 5 09:36:58 2018 +0900 Merge branch 'master-ce' into stateful_deployments commit b34d165320d6f3298c8b776ba66270a59c217412 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 18:07:08 2018 +0900 Fix flaky spec commit b5e0527c5d4fe8f18b2fdda5916bae9b8cd859a4 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 15:32:03 2018 +0900 Fix spec commit f78a5e96e66fe2d25086df495e339b470a274df8 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 14:59:29 2018 +0900 Remove unnecessary line in schema.rb commit 6ce7c483e0591b5d6f9588a99853834327b80031 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 14:55:48 2018 +0900 Add partial index for filling deployment at migration commit aecccfb5118c8982db3ba502fdf37b5e639fbfc6 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 14:42:24 2018 +0900 Fix fill empty finished at migration commit 0199e1761ad1b391ae87a53a9a113d3256529e0e Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 14:19:44 2018 +0900 Fix flaky spec commit 56ac84cd8095afab5b909119445537b7da06a2ff Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 10:06:49 2018 +0900 Fix guard clause to prevent multiple deployments to a job commit 521561b6b303b54635c30cb23d78e49d14cec53d Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 20:19:24 2018 +0900 Fix spec commit 2878da0d29b9bd2dde69a1b216203df118dd59a1 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 19:38:59 2018 +0900 Simplify the factory commit 22fd7df02133f3a21828554965fd5619905eac2c Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 19:33:50 2018 +0900 Simplify the Deployable and BuildSuccessWorker commit 41108959677ed614f4548443a2f4303c4c04925a Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 18:34:20 2018 +0900 Fix spec commit ae75fe7461ac72f621498797f478d42331342b84 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 17:19:12 2018 +0900 Fix weird virtual deployment status commit 380fee7494d06407dccc292c3cbedbcee7b6e235 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 15:59:31 2018 +0900 Fix spec commit 29889fcbaadb3bbfd2f11c10bfbf5dceb3e3ddba Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 15:07:10 2018 +0900 Fix coding offence commit 36ac13f345f5ef25725c2236a791a40a3a9e6126 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 14:22:17 2018 +0900 Squashed commit of the following: commit ba9aede922e1643db3f06c56736d46d6d86d356b Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 14:21:33 2018 +0900 Fix ambiguious factory specification in update deployment service spec commit 013afb5668cb30dc4ca5b21945c17b341e7ea7f9 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 14:10:24 2018 +0900 Fix spec commit 78793670d049e2dfb5fc98177eb4d10f20b9310b Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 18:26:12 2018 +0900 Fix spec commit 73d27e87c66698f2e3a817bb8728f02475b7ba4f Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 16:22:14 2018 +0900 Fix index commit 8580a226ea68bf5e49b35bfb5f404968bbfaf8e9 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 15:34:57 2018 +0900 Fix deployment relationships in Ci::Build commit d6d28b55afd1179200b4f5188e0b53079ff3c1a7 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 15:27:53 2018 +0900 Fix spec commit 94eb754e2e1bb9a1fe627f86823f571a8298d27b Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 14:07:11 2018 +0900 Fix spec commit 0b30f80bcd08a7a06bdde3378ec1733f865284be Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 20:15:31 2018 +0900 Fix spec commit 466bdcdb6af8cdb475c9fa16bd7d1dff23b11e40 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 19:28:51 2018 +0900 Fix spec commit a7c3caac99139e70fe3f1f3d14856939fa25c527 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 17:33:47 2018 +0900 Fix factory commit cea28ae100532e6711ce1d22676719a94e2da8a0 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 16:28:18 2018 +0900 Drop leagacy success commit 3785d685eabc10b6597cf3db67bf08385ccf298a Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:37:28 2018 +0900 Remove unnecessary migration file commit 0d597fa46eeffdbb9a4afb53005a8183e433c6bf Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:35:53 2018 +0900 Fix schema.rb commit ec3c2abc6944e09f6410468ae5e356865ec7b02b Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:34:21 2018 +0900 Rename post migration file commit 0e7281885a84656acf95f0f423732680f8fec076 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:31:01 2018 +0900 Remove include EnumWithNil commit b3846d59c07e07275126c70361bde7f30810729e Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:05:50 2018 +0900 Decouple action commit c9f9ba4eae9ca1edc7d8751e1d2e0572cb222d9c Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 14:23:29 2018 +0900 Remove status mock commit d95bfea1ca67b3a27a3226a669c2b1266d696682 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 14:17:14 2018 +0900 Add action commit 0cec39e0f76c22a18498f46d65ad7226fb30c3f8 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 13:44:07 2018 +0900 Remove unnecessary line in schema.rb commit 7b4c5f8e1b00dd8e6aa944352f9d8a9f3ae6f1c7 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 19:59:41 2018 +0900 Revert build success worker commit 0c52ffa4a23eea488c187317e8b400369846f399 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 19:11:47 2018 +0900 Use add_column_with_default properly commit ba9bae357da5dfd2f6ec05f7f9db9d0b31224f48 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 18:40:55 2018 +0900 Fix with_status commit 75dffc97b9c5f6fa73d9d09b125c8f849fa2caae Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 16:26:56 2018 +0900 Remove unnecessary line in schema.rb commit 25188ccc52fb29ca63b9205c4d95ffc2e0afadee Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 16:26:17 2018 +0900 Set default values in regular migration commit 98ea037fbf39c8d9f0db77fb50e2d08382425158 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 17:27:49 2018 +0900 Fix static analysis commit e7d1765f77f9ff9b94a34985a7855bdaab1da675 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 16:37:10 2018 +0900 Remove empty spec commit 0033f521ed1eae8117dba231961aa47c068bbcfb Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 16:34:55 2018 +0900 Simplify spec changes commit 0be4c6b3ade6d9a8bf28bcd177c66ebd7bb7d20a Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 16:32:45 2018 +0900 Simplify spec changes commit a93d25d79df7e25bdf688fc938c712922f9dc4df Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 16:02:31 2018 +0900 Fix flaky spec commit 339ad50cf471ca706b29f008ccd2bb881dd5b776 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 15:06:22 2018 +0900 Rename Deployments Success worker commit bd69c78085adcb9b0f8ff9b7041ae355953ad7ab Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 14:43:03 2018 +0900 Fix coding offence commit 004748b2a9c5236ec13eb01289418f3d6571c92c Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 20:09:10 2018 +0900 Rename to update deployment service commit b04a85e761de501f030f3844fd485a2b9e46f7f7 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 18:46:52 2018 +0900 Add spec for Project commit 548af23a5a07f0c20b72849d03aa0b98a0b49134 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 18:43:25 2018 +0900 Fix spec commit c977e4d3f17194c46a1bf857b473017ce21ef7e9 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 17:58:07 2018 +0900 Add spec for Environment commit 73feb9010f8d8093bee4b46e56d30cfef3e8e34a Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 17:39:24 2018 +0900 Add spec for Deployment model commit 9a3cfbf766f402571588839375cf311bb9807035 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 17:18:02 2018 +0900 Fix statis analysis commit a30d28dbc631a29855883ca89c592a10c012f1d2 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 17:17:32 2018 +0900 Ignore nil instance commit fa6fdd89f380e588a6bcf14b1f9aef0d14d3854b Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 16:20:40 2018 +0900 Add spec for deployable concern commit aa91186821dc671df2c7a641e37586dd5dfc1008 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 15:37:23 2018 +0900 Clean up deployable commit 34d3e18731f7906a3db250b105a64d1db83c2fca Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 15:13:05 2018 +0900 Fix 17 cycle analytics commit 8dc9e00408f9b390175e7d5ea743eed4fb9e3f79 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 13:56:51 2018 +0900 Fix static analysys commit 5c4175807a537bafc4b889b0a97e8f96f0e483cd Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 24 15:05:05 2018 +0900 Skip unnecessary sidekiq worker commit 9d8b5d423f49cc247c96ce3767d03b4af305809f Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 24 14:53:13 2018 +0900 Add changelog commit c8cabba496722240cadf7c161c80bceb09727cba Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 24 14:44:45 2018 +0900 Squashed commit of the following: commit f7643885ac2329e18d690a4e4f2d7614b732c793 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 19:38:45 2018 +0900 Fix deployment widget specs commit 03bd04b5c98b634dff6a0ab4292c150a9031995c Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 17:50:16 2018 +0900 Fix env status spec commit 4a49c6502b161a12f0f62d5ec167dff777047dab Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 17:48:59 2018 +0900 Fix environment spec commit 4044822887987e20a703990ff20352a532eeb965 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 17:47:17 2018 +0900 Fix environment spec commit 9939d44b7eb9da371de74c0f04fed1eb3db37ad3 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 15:45:43 2018 +0900 Add a new spec for deployment success worker commit f61c4d3657b5ef13b5da171460da68a6643ad4b5 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 15:38:11 2018 +0900 Fix cycle analytics helper commit b6242615e8298fb7fc047c8df8006c25ad717c70 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 14:41:54 2018 +0900 Fix cycle analysis helper commit 9a001cb4c4ed6f3b87dc612bdffc60a6b2b0a132 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 14:37:08 2018 +0900 Ignore coding offence in build success worker's spec commit 1fb88583025bac8a56172cbd59be04258ea4c5f3 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 14:33:11 2018 +0900 Added more spec for deployments commit 1a6ba97ababbf62e8dd0ae0c56d75ab1268fd0ce Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 19:36:50 2018 +0900 Move after create hookd into success worker commit 09de5fed5d6f108423779cf9d9e7f1d21f3c1c91 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 19:30:06 2018 +0900 Fix build spec commit 73a55cbcabbb1e928eca3e53e8ff75dec178bc90 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 19:08:43 2018 +0900 Fix update_deployment_metrics_service_spec.rb commit ee05136a02ae9fa348b4b89b9a69937ebb9697dd Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 17:32:05 2018 +0900 Remove unnecessary degelate commit e246ddeebc01a807ccc36fdb484c3e72ad91e680 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 16:07:39 2018 +0900 Remove unnecessary optimistic locking commit dcc225c8237b90e3bc8dcc3dc2e3252e0b0be093 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 16:00:22 2018 +0900 Simplify status replication commit 13a5fd7afb67ba2712fcaecaea5fedf05f9ad177 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 15:29:24 2018 +0900 Fix sidekiq queue names commit dcc796f48d523538e1c91b9cd3e1c7065e5329b1 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 15:23:55 2018 +0900 Revert success check in update_merge_request_metrics commit 129ef083d637d4acb8c97a6d9ab96deb2ff6efcd Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 15:18:31 2018 +0900 Fix queue name of deployment success worker commit 10fe5a6484f4f02322ce5bb16844fc7b1d565963 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 15:09:42 2018 +0900 Introduce deployable module commit d91260bbe105bf46f6c06d9e9593c8c4cd5139cf Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 14:05:31 2018 +0900 Add database index for successful deployments commit 74274147263de4b60870065a19935498ce662e30 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 13:51:59 2018 +0900 Fix invalid state transition commit ff18463cc847bf3cf5a3e49f3651eedfdf67c7e6 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 20:05:15 2018 +0900 Fix coding style offence commit 0202c0f5b631601edab7b359b087b307f5eb7ba3 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 18:34:07 2018 +0900 Target only successful deployments from other relations commit 1f2758cb030dec1df5dda30f6bc3e25b6d0841c9 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 18:21:28 2018 +0900 Add namespace explicitly commit 3d9227b6e5642cecde88d4edac925125f6474b11 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 17:42:30 2018 +0900 Fix spec in DeleteInconsistentInternalIdRecords commit 3e0cc99ff6c5c7188511618228a6ec027752ce69 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 16:10:16 2018 +0900 Fixed spec commit 8de09b8bb31f7b9f24ecdf9f2dd8ef358a260263 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 14:22:35 2018 +0900 Fix create deployment service commit 31957570b4444492eeb412e765f96a56416c25f3 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 20:21:26 2018 +0900 Move CreateDeploymentService. Fix Cycle analytics spec and fixture. commit d2eb433a1bb9710c0d4778c4f34c12b6b64f60e6 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 20:11:22 2018 +0900 Fix build success worker commit 25e6cd87138bcdb69de8785ca367e479c8dbcc59 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 19:49:13 2018 +0900 Fix create deployment service spec commit d268bf410bf65e86c81eb76d50aa8e145b32d249 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 19:01:23 2018 +0900 Fix cycle analysys spec's deployment commit 525ade8aa1e4394ed8a759bb0437e407fbe74a35 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 18:24:04 2018 +0900 Fix factory to set legacy status by default commit c6a990821ac0a1ffa49e20e2d78d94b8ce075914 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 17:25:40 2018 +0900 Remove unnecessary lib from deployment commit a6107e0e85ac26ee09da3316ebc11de32f067d82 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 17:38:58 2018 +0900 Fix recursive call commit 15c5f3b64061a75af3c3039ca7f49b1cc4ff3068 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 17:30:44 2018 +0900 Add finished_at commit c8d3d70366f694d78acb7e30d342c7697798b922 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 15:55:31 2018 +0900 Fix last_deployment methods as it used to return successful deployment always commit 96bbe8670cece021766fde95fe573cbbe23d1e55 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 15:49:57 2018 +0900 Redefine statuses commit c86a9d0bd2ab3e7a00bf61f094a96ee99b76b289 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 14:50:27 2018 +0900 Fix schema.rb commit 9ff5f0eaafbc08795018c7bb282b19f6327dee21 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 14:18:04 2018 +0900 Default status nil to success commit 5928bd9bb94e1e8908ed1561e01595be84d5f4ec Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 16 15:13:48 2018 +0900 Add status to Deployment
2018-11-04 19:37:40 -05:00
pipeline.builds.map(&:run!)
pipeline.update_status
end
end
def merge_merge_requests(merge_requests)
merge_requests.each do |merge_request|
Timecop.travel 12.hours.from_now
MergeRequests::MergeService.new(merge_request.project, @user).execute(merge_request)
end
end
def deploy_to_production(merge_requests)
merge_requests.each do |merge_request|
next unless merge_request.head_pipeline
Timecop.travel 12.hours.from_now
job = merge_request.head_pipeline.builds.where.not(environment: nil).last
Squashed commit of the following: commit 931d6ab0e025b0268d94e455f736b09a025e0578 Merge: b34d165320d 93846eb152f Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Nov 5 09:36:58 2018 +0900 Merge branch 'master-ce' into stateful_deployments commit b34d165320d6f3298c8b776ba66270a59c217412 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 18:07:08 2018 +0900 Fix flaky spec commit b5e0527c5d4fe8f18b2fdda5916bae9b8cd859a4 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 15:32:03 2018 +0900 Fix spec commit f78a5e96e66fe2d25086df495e339b470a274df8 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 14:59:29 2018 +0900 Remove unnecessary line in schema.rb commit 6ce7c483e0591b5d6f9588a99853834327b80031 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 14:55:48 2018 +0900 Add partial index for filling deployment at migration commit aecccfb5118c8982db3ba502fdf37b5e639fbfc6 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 14:42:24 2018 +0900 Fix fill empty finished at migration commit 0199e1761ad1b391ae87a53a9a113d3256529e0e Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 14:19:44 2018 +0900 Fix flaky spec commit 56ac84cd8095afab5b909119445537b7da06a2ff Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 10:06:49 2018 +0900 Fix guard clause to prevent multiple deployments to a job commit 521561b6b303b54635c30cb23d78e49d14cec53d Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 20:19:24 2018 +0900 Fix spec commit 2878da0d29b9bd2dde69a1b216203df118dd59a1 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 19:38:59 2018 +0900 Simplify the factory commit 22fd7df02133f3a21828554965fd5619905eac2c Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 19:33:50 2018 +0900 Simplify the Deployable and BuildSuccessWorker commit 41108959677ed614f4548443a2f4303c4c04925a Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 18:34:20 2018 +0900 Fix spec commit ae75fe7461ac72f621498797f478d42331342b84 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 17:19:12 2018 +0900 Fix weird virtual deployment status commit 380fee7494d06407dccc292c3cbedbcee7b6e235 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 15:59:31 2018 +0900 Fix spec commit 29889fcbaadb3bbfd2f11c10bfbf5dceb3e3ddba Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 15:07:10 2018 +0900 Fix coding offence commit 36ac13f345f5ef25725c2236a791a40a3a9e6126 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 14:22:17 2018 +0900 Squashed commit of the following: commit ba9aede922e1643db3f06c56736d46d6d86d356b Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 14:21:33 2018 +0900 Fix ambiguious factory specification in update deployment service spec commit 013afb5668cb30dc4ca5b21945c17b341e7ea7f9 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 14:10:24 2018 +0900 Fix spec commit 78793670d049e2dfb5fc98177eb4d10f20b9310b Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 18:26:12 2018 +0900 Fix spec commit 73d27e87c66698f2e3a817bb8728f02475b7ba4f Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 16:22:14 2018 +0900 Fix index commit 8580a226ea68bf5e49b35bfb5f404968bbfaf8e9 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 15:34:57 2018 +0900 Fix deployment relationships in Ci::Build commit d6d28b55afd1179200b4f5188e0b53079ff3c1a7 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 15:27:53 2018 +0900 Fix spec commit 94eb754e2e1bb9a1fe627f86823f571a8298d27b Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 14:07:11 2018 +0900 Fix spec commit 0b30f80bcd08a7a06bdde3378ec1733f865284be Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 20:15:31 2018 +0900 Fix spec commit 466bdcdb6af8cdb475c9fa16bd7d1dff23b11e40 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 19:28:51 2018 +0900 Fix spec commit a7c3caac99139e70fe3f1f3d14856939fa25c527 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 17:33:47 2018 +0900 Fix factory commit cea28ae100532e6711ce1d22676719a94e2da8a0 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 16:28:18 2018 +0900 Drop leagacy success commit 3785d685eabc10b6597cf3db67bf08385ccf298a Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:37:28 2018 +0900 Remove unnecessary migration file commit 0d597fa46eeffdbb9a4afb53005a8183e433c6bf Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:35:53 2018 +0900 Fix schema.rb commit ec3c2abc6944e09f6410468ae5e356865ec7b02b Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:34:21 2018 +0900 Rename post migration file commit 0e7281885a84656acf95f0f423732680f8fec076 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:31:01 2018 +0900 Remove include EnumWithNil commit b3846d59c07e07275126c70361bde7f30810729e Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:05:50 2018 +0900 Decouple action commit c9f9ba4eae9ca1edc7d8751e1d2e0572cb222d9c Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 14:23:29 2018 +0900 Remove status mock commit d95bfea1ca67b3a27a3226a669c2b1266d696682 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 14:17:14 2018 +0900 Add action commit 0cec39e0f76c22a18498f46d65ad7226fb30c3f8 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 13:44:07 2018 +0900 Remove unnecessary line in schema.rb commit 7b4c5f8e1b00dd8e6aa944352f9d8a9f3ae6f1c7 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 19:59:41 2018 +0900 Revert build success worker commit 0c52ffa4a23eea488c187317e8b400369846f399 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 19:11:47 2018 +0900 Use add_column_with_default properly commit ba9bae357da5dfd2f6ec05f7f9db9d0b31224f48 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 18:40:55 2018 +0900 Fix with_status commit 75dffc97b9c5f6fa73d9d09b125c8f849fa2caae Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 16:26:56 2018 +0900 Remove unnecessary line in schema.rb commit 25188ccc52fb29ca63b9205c4d95ffc2e0afadee Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 16:26:17 2018 +0900 Set default values in regular migration commit 98ea037fbf39c8d9f0db77fb50e2d08382425158 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 17:27:49 2018 +0900 Fix static analysis commit e7d1765f77f9ff9b94a34985a7855bdaab1da675 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 16:37:10 2018 +0900 Remove empty spec commit 0033f521ed1eae8117dba231961aa47c068bbcfb Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 16:34:55 2018 +0900 Simplify spec changes commit 0be4c6b3ade6d9a8bf28bcd177c66ebd7bb7d20a Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 16:32:45 2018 +0900 Simplify spec changes commit a93d25d79df7e25bdf688fc938c712922f9dc4df Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 16:02:31 2018 +0900 Fix flaky spec commit 339ad50cf471ca706b29f008ccd2bb881dd5b776 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 15:06:22 2018 +0900 Rename Deployments Success worker commit bd69c78085adcb9b0f8ff9b7041ae355953ad7ab Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 14:43:03 2018 +0900 Fix coding offence commit 004748b2a9c5236ec13eb01289418f3d6571c92c Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 20:09:10 2018 +0900 Rename to update deployment service commit b04a85e761de501f030f3844fd485a2b9e46f7f7 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 18:46:52 2018 +0900 Add spec for Project commit 548af23a5a07f0c20b72849d03aa0b98a0b49134 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 18:43:25 2018 +0900 Fix spec commit c977e4d3f17194c46a1bf857b473017ce21ef7e9 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 17:58:07 2018 +0900 Add spec for Environment commit 73feb9010f8d8093bee4b46e56d30cfef3e8e34a Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 17:39:24 2018 +0900 Add spec for Deployment model commit 9a3cfbf766f402571588839375cf311bb9807035 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 17:18:02 2018 +0900 Fix statis analysis commit a30d28dbc631a29855883ca89c592a10c012f1d2 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 17:17:32 2018 +0900 Ignore nil instance commit fa6fdd89f380e588a6bcf14b1f9aef0d14d3854b Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 16:20:40 2018 +0900 Add spec for deployable concern commit aa91186821dc671df2c7a641e37586dd5dfc1008 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 15:37:23 2018 +0900 Clean up deployable commit 34d3e18731f7906a3db250b105a64d1db83c2fca Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 15:13:05 2018 +0900 Fix 17 cycle analytics commit 8dc9e00408f9b390175e7d5ea743eed4fb9e3f79 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 13:56:51 2018 +0900 Fix static analysys commit 5c4175807a537bafc4b889b0a97e8f96f0e483cd Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 24 15:05:05 2018 +0900 Skip unnecessary sidekiq worker commit 9d8b5d423f49cc247c96ce3767d03b4af305809f Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 24 14:53:13 2018 +0900 Add changelog commit c8cabba496722240cadf7c161c80bceb09727cba Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 24 14:44:45 2018 +0900 Squashed commit of the following: commit f7643885ac2329e18d690a4e4f2d7614b732c793 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 19:38:45 2018 +0900 Fix deployment widget specs commit 03bd04b5c98b634dff6a0ab4292c150a9031995c Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 17:50:16 2018 +0900 Fix env status spec commit 4a49c6502b161a12f0f62d5ec167dff777047dab Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 17:48:59 2018 +0900 Fix environment spec commit 4044822887987e20a703990ff20352a532eeb965 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 17:47:17 2018 +0900 Fix environment spec commit 9939d44b7eb9da371de74c0f04fed1eb3db37ad3 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 15:45:43 2018 +0900 Add a new spec for deployment success worker commit f61c4d3657b5ef13b5da171460da68a6643ad4b5 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 15:38:11 2018 +0900 Fix cycle analytics helper commit b6242615e8298fb7fc047c8df8006c25ad717c70 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 14:41:54 2018 +0900 Fix cycle analysis helper commit 9a001cb4c4ed6f3b87dc612bdffc60a6b2b0a132 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 14:37:08 2018 +0900 Ignore coding offence in build success worker's spec commit 1fb88583025bac8a56172cbd59be04258ea4c5f3 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 14:33:11 2018 +0900 Added more spec for deployments commit 1a6ba97ababbf62e8dd0ae0c56d75ab1268fd0ce Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 19:36:50 2018 +0900 Move after create hookd into success worker commit 09de5fed5d6f108423779cf9d9e7f1d21f3c1c91 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 19:30:06 2018 +0900 Fix build spec commit 73a55cbcabbb1e928eca3e53e8ff75dec178bc90 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 19:08:43 2018 +0900 Fix update_deployment_metrics_service_spec.rb commit ee05136a02ae9fa348b4b89b9a69937ebb9697dd Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 17:32:05 2018 +0900 Remove unnecessary degelate commit e246ddeebc01a807ccc36fdb484c3e72ad91e680 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 16:07:39 2018 +0900 Remove unnecessary optimistic locking commit dcc225c8237b90e3bc8dcc3dc2e3252e0b0be093 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 16:00:22 2018 +0900 Simplify status replication commit 13a5fd7afb67ba2712fcaecaea5fedf05f9ad177 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 15:29:24 2018 +0900 Fix sidekiq queue names commit dcc796f48d523538e1c91b9cd3e1c7065e5329b1 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 15:23:55 2018 +0900 Revert success check in update_merge_request_metrics commit 129ef083d637d4acb8c97a6d9ab96deb2ff6efcd Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 15:18:31 2018 +0900 Fix queue name of deployment success worker commit 10fe5a6484f4f02322ce5bb16844fc7b1d565963 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 15:09:42 2018 +0900 Introduce deployable module commit d91260bbe105bf46f6c06d9e9593c8c4cd5139cf Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 14:05:31 2018 +0900 Add database index for successful deployments commit 74274147263de4b60870065a19935498ce662e30 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 13:51:59 2018 +0900 Fix invalid state transition commit ff18463cc847bf3cf5a3e49f3651eedfdf67c7e6 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 20:05:15 2018 +0900 Fix coding style offence commit 0202c0f5b631601edab7b359b087b307f5eb7ba3 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 18:34:07 2018 +0900 Target only successful deployments from other relations commit 1f2758cb030dec1df5dda30f6bc3e25b6d0841c9 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 18:21:28 2018 +0900 Add namespace explicitly commit 3d9227b6e5642cecde88d4edac925125f6474b11 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 17:42:30 2018 +0900 Fix spec in DeleteInconsistentInternalIdRecords commit 3e0cc99ff6c5c7188511618228a6ec027752ce69 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 16:10:16 2018 +0900 Fixed spec commit 8de09b8bb31f7b9f24ecdf9f2dd8ef358a260263 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 14:22:35 2018 +0900 Fix create deployment service commit 31957570b4444492eeb412e765f96a56416c25f3 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 20:21:26 2018 +0900 Move CreateDeploymentService. Fix Cycle analytics spec and fixture. commit d2eb433a1bb9710c0d4778c4f34c12b6b64f60e6 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 20:11:22 2018 +0900 Fix build success worker commit 25e6cd87138bcdb69de8785ca367e479c8dbcc59 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 19:49:13 2018 +0900 Fix create deployment service spec commit d268bf410bf65e86c81eb76d50aa8e145b32d249 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 19:01:23 2018 +0900 Fix cycle analysys spec's deployment commit 525ade8aa1e4394ed8a759bb0437e407fbe74a35 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 18:24:04 2018 +0900 Fix factory to set legacy status by default commit c6a990821ac0a1ffa49e20e2d78d94b8ce075914 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 17:25:40 2018 +0900 Remove unnecessary lib from deployment commit a6107e0e85ac26ee09da3316ebc11de32f067d82 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 17:38:58 2018 +0900 Fix recursive call commit 15c5f3b64061a75af3c3039ca7f49b1cc4ff3068 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 17:30:44 2018 +0900 Add finished_at commit c8d3d70366f694d78acb7e30d342c7697798b922 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 15:55:31 2018 +0900 Fix last_deployment methods as it used to return successful deployment always commit 96bbe8670cece021766fde95fe573cbbe23d1e55 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 15:49:57 2018 +0900 Redefine statuses commit c86a9d0bd2ab3e7a00bf61f094a96ee99b76b289 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 14:50:27 2018 +0900 Fix schema.rb commit 9ff5f0eaafbc08795018c7bb282b19f6327dee21 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 14:18:04 2018 +0900 Default status nil to success commit 5928bd9bb94e1e8908ed1561e01595be84d5f4ec Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 16 15:13:48 2018 +0900 Add status to Deployment
2018-11-04 19:37:40 -05:00
job.success!
pipeline.update_status
end
end
end
Gitlab::Seeder.quiet do
flag = 'SEED_CYCLE_ANALYTICS'
if ENV[flag]
Project.find_each do |project|
# This seed naively assumes that every project has a repository, and every
# repository has a `master` branch, which may be the case for a pristine
# GDK seed, but is almost never true for a GDK that's actually had
# development performed on it.
next unless project.repository_exists?
next unless project.repository.commit('master')
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
seeder = Gitlab::Seeder::CycleAnalytics.new(project)
2016-09-14 10:10:31 -04:00
seeder.seed!
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
end
2016-09-14 10:10:31 -04:00
elsif ENV['CYCLE_ANALYTICS_PERF_TEST']
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
seeder = Gitlab::Seeder::CycleAnalytics.new(Project.order(:id).first, perf: true)
seeder.seed!
elsif ENV['CYCLE_ANALYTICS_POPULATE_METRICS_DIRECTLY']
seeder = Gitlab::Seeder::CycleAnalytics.new(Project.order(:id).first, perf: true)
seeder.seed_metrics!
2016-09-14 10:10:31 -04:00
else
puts "Skipped. Use the `#{flag}` environment variable to enable."
end
end