gitlab-org--gitlab-foss/app/services/merge_requests/refresh_service.rb

224 lines
7.7 KiB
Ruby
Raw Normal View History

# frozen_string_literal: true
module MergeRequests
class RefreshService < MergeRequests::BaseService
def execute(oldrev, newrev, ref)
return true unless Gitlab::Git.branch_ref?(ref)
do_execute(oldrev, newrev, ref)
end
private
def do_execute(oldrev, newrev, ref)
@oldrev, @newrev = oldrev, newrev
@branch_name = Gitlab::Git.ref_name(ref)
2015-10-21 11:34:12 -04:00
Gitlab::GitalyClient.allow_n_plus_1_calls(&method(:find_new_commits))
# Be sure to close outstanding MRs before reloading them to avoid generating an
# empty diff during a manual merge
close_upon_missing_source_branch_ref
post_merge_manually_merged
reload_merge_requests
reset_merge_when_pipeline_succeeds
mark_pending_todos_done
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
cache_merge_requests_closing_issues
2015-10-21 11:34:12 -04:00
# Leave a system note if a branch was deleted/added
if branch_added? || branch_removed?
2015-10-16 01:45:06 -04:00
comment_mr_branch_presence_changed
end
notify_about_push
mark_mr_as_wip_from_commits
execute_mr_web_hooks
true
end
def close_upon_missing_source_branch_ref
# MergeRequest#reload_diff ignores not opened MRs. This means it won't
# create an `empty` diff for `closed` MRs without a source branch, keeping
# the latest diff state as the last _valid_ one.
merge_requests_for_source_branch.reject(&:source_branch_exists?).each do |mr|
MergeRequests::CloseService
.new(mr.target_project, @current_user)
.execute(mr)
end
end
# Collect open merge requests that target same branch we push into
# and close if push to master include last commit from merge request
# We need this to close(as merged) merge requests that were merged into
# target branch manually
def post_merge_manually_merged
commit_ids = @commits.map(&:id)
Use latest_merge_request_diff association Compared to the merge_request_diff association: 1. It's simpler to query. The query uses a foreign key to the merge_request_diffs table, so no ordering is necessary. 2. It's faster for preloading. The merge_request_diff association has to load every diff for the MRs in the set, then discard all but the most recent for each. This association means that Rails can just query for N diffs from N MRs. 3. It's more complicated to update. This is a bidirectional foreign key, so we need to update two tables when adding a diff record. This also means we need to handle this as a special case when importing a GitLab project. There is some juggling with this association in the merge request model: * `MergeRequest#latest_merge_request_diff` is _always_ the latest diff. * `MergeRequest#merge_request_diff` reuses `MergeRequest#latest_merge_request_diff` unless: * Arguments are passed. These are typically to force-reload the association. * It doesn't exist. That means we might be trying to implicitly create a diff. This only seems to happen in specs. * The association is already loaded. This is important for the reasons explained in the comment, which I'll reiterate here: if we a) load a non-latest diff, then b) get its `merge_request`, then c) get that MR's `merge_request_diff`, we should get the diff we loaded in c), even though that's not the latest diff. Basically, `MergeRequest#merge_request_diff` is the latest diff in most cases, but not quite all.
2017-11-15 12:22:18 -05:00
merge_requests = @project.merge_requests.preload(:latest_merge_request_diff).opened.where(target_branch: @branch_name).to_a
merge_requests = merge_requests.select(&:diff_head_commit)
merge_requests = merge_requests.select do |merge_request|
commit_ids.include?(merge_request.diff_head_sha) &&
merge_request.merge_request_diff.state != 'empty'
end
filter_merge_requests(merge_requests).each do |merge_request|
2017-06-21 09:48:12 -04:00
MergeRequests::PostMergeService
.new(merge_request.target_project, @current_user)
.execute(merge_request)
end
end
def force_push?
Gitlab::Checks::ForcePush.force_push?(@project, @oldrev, @newrev)
end
# Refresh merge request diff if we push to source or target branch of merge request
# Note: we should update merge requests from forks too
def reload_merge_requests
2017-06-21 09:48:12 -04:00
merge_requests = @project.merge_requests.opened
.by_source_or_target_branch(@branch_name).to_a
2015-10-15 04:57:45 -04:00
# Fork merge requests
merge_requests += MergeRequest.opened
.where(source_branch: @branch_name, source_project: @project)
.where.not(target_project: @project).to_a
filter_merge_requests(merge_requests).each do |merge_request|
if merge_request.source_branch == @branch_name || force_push?
merge_request.reload_diff(current_user)
else
mr_commit_ids = merge_request.commit_shas
push_commit_ids = @commits.map(&:id)
matches = mr_commit_ids & push_commit_ids
merge_request.reload_diff(current_user) if matches.any?
end
merge_request.mark_as_unchecked
UpdateHeadPipelineForMergeRequestWorker.perform_async(merge_request.id)
end
# Upcoming method calls need the refreshed version of
# @source_merge_requests diffs (for MergeRequest#commit_shas for instance).
merge_requests_for_source_branch(reload: true)
end
def reset_merge_when_pipeline_succeeds
merge_requests_for_source_branch.each(&:reset_merge_when_pipeline_succeeds)
2015-11-02 11:27:38 -05:00
end
def mark_pending_todos_done
merge_requests_for_source_branch.each do |merge_request|
todo_service.merge_request_push(merge_request, @current_user)
end
end
2015-10-21 11:34:12 -04:00
def find_new_commits
if branch_added?
@commits = []
2015-10-16 01:45:06 -04:00
2015-10-21 11:34:12 -04:00
merge_request = merge_requests_for_source_branch.first
return unless merge_request
2015-10-16 01:45:06 -04:00
2015-10-21 11:34:12 -04:00
begin
# Since any number of commits could have been made to the restored branch,
# find the common root to see what has been added.
common_ref = @project.repository.merge_base(merge_request.diff_head_sha, @newrev)
2015-10-21 11:34:12 -04:00
# If the a commit no longer exists in this repo, gitlab_git throws
# a Rugged::OdbError. This is fixed in https://gitlab.com/gitlab-org/gitlab_git/merge_requests/52
@commits = @project.repository.commits_between(common_ref, @newrev) if common_ref
rescue
2015-10-16 01:45:06 -04:00
end
2015-10-21 11:34:12 -04:00
elsif branch_removed?
# No commits for a deleted branch.
@commits = []
else
@commits = @project.repository.commits_between(@oldrev, @newrev)
end
end
2015-10-16 01:45:06 -04:00
2015-10-21 11:34:12 -04:00
# Add comment about branches being deleted or added to merge requests
def comment_mr_branch_presence_changed
presence = branch_added? ? :add : :delete
merge_requests_for_source_branch.each do |merge_request|
SystemNoteService.change_branch_presence(
2015-12-14 21:53:52 -05:00
merge_request, merge_request.project, @current_user,
:source, @branch_name, presence)
end
end
# Add comment about pushing new commits to merge requests and send nofitication emails
def notify_about_push
2015-10-21 11:34:12 -04:00
return unless @commits.present?
2015-10-15 05:24:30 -04:00
merge_requests_for_source_branch.each do |merge_request|
mr_commit_ids = Set.new(merge_request.commit_shas)
new_commits, existing_commits = @commits.partition do |commit|
mr_commit_ids.include?(commit.id)
end
SystemNoteService.add_commits(merge_request, merge_request.project,
@current_user, new_commits,
existing_commits, @oldrev)
notification_service.push_to_merge_request(merge_request, @current_user, new_commits: new_commits, existing_commits: existing_commits)
end
end
def mark_mr_as_wip_from_commits
return unless @commits.present?
merge_requests_for_source_branch.each do |merge_request|
commit_shas = merge_request.commit_shas
2017-02-09 05:34:02 -05:00
wip_commit = @commits.detect do |commit|
commit.work_in_progress? && commit_shas.include?(commit.sha)
end
if wip_commit && !merge_request.work_in_progress?
merge_request.update(title: merge_request.wip_title)
SystemNoteService.add_merge_request_wip_from_commit(
merge_request,
merge_request.project,
@current_user,
wip_commit
)
end
end
end
# Call merge request webhook with update branches
def execute_mr_web_hooks
2015-10-15 05:24:30 -04:00
merge_requests_for_source_branch.each do |merge_request|
execute_hooks(merge_request, 'update', old_rev: @oldrev)
end
end
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
# If the merge requests closes any issues, save this information in the
# `MergeRequestsClosingIssues` model (as a performance optimization).
def cache_merge_requests_closing_issues
@project.merge_requests.where(source_branch: @branch_name).each do |merge_request|
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
merge_request.cache_merge_request_closes_issues!(@current_user)
end
end
def filter_merge_requests(merge_requests)
merge_requests.uniq.select(&:source_project)
end
def merge_requests_for_source_branch(reload: false)
@source_merge_requests = nil if reload
@source_merge_requests ||= merge_requests_for(@branch_name)
2015-10-21 11:34:12 -04:00
end
def branch_added?
Gitlab::Git.blank_ref?(@oldrev)
end
def branch_removed?
Gitlab::Git.blank_ref?(@newrev)
end
end
end