From ba9b4c4de86aa816e5ddc7a9cde9193c43835223 Mon Sep 17 00:00:00 2001 From: Sean McGivern Date: Fri, 27 Oct 2017 14:20:55 +0100 Subject: [PATCH] Avoid hitting statement timeout finding MR pipelines For MRs with many thousands of commits, `SELECT DISTINCT(sha)` will be very slow. What we can't do to fix this: 1. Add an index. Postgres won't use it for DISTINCT without a lot of ceremony. 2. Do the `uniq` in Ruby. That can still be very slow with hundreds of thousands of commits. 3. Use a subquery. We haven't removed the `st_commits` column yet, but we will soon. Until 3 is available to us, we can just do 2, but also add a limit clause. There is no ordering, so this may return different results, but our goal with these MRs is just to get them to load, so it's not a huge deal. --- app/models/merge_request.rb | 2 +- ...ed-error-canceling-statement-due-to-statement-timeout.yml | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) create mode 100644 changelogs/unreleased/39054-activerecord-statementinvalid-pg-querycanceled-error-canceling-statement-due-to-statement-timeout.yml diff --git a/app/models/merge_request.rb b/app/models/merge_request.rb index c3fae16d109..b0a0c753c09 100644 --- a/app/models/merge_request.rb +++ b/app/models/merge_request.rb @@ -874,7 +874,7 @@ class MergeRequest < ActiveRecord::Base # def all_commit_shas if persisted? - column_shas = MergeRequestDiffCommit.where(merge_request_diff: merge_request_diffs).pluck('DISTINCT(sha)') + column_shas = MergeRequestDiffCommit.where(merge_request_diff: merge_request_diffs).limit(10_000).pluck('sha') serialised_shas = merge_request_diffs.where.not(st_commits: nil).flat_map(&:commit_shas) (column_shas + serialised_shas).uniq diff --git a/changelogs/unreleased/39054-activerecord-statementinvalid-pg-querycanceled-error-canceling-statement-due-to-statement-timeout.yml b/changelogs/unreleased/39054-activerecord-statementinvalid-pg-querycanceled-error-canceling-statement-due-to-statement-timeout.yml new file mode 100644 index 00000000000..47bf30ecb5a --- /dev/null +++ b/changelogs/unreleased/39054-activerecord-statementinvalid-pg-querycanceled-error-canceling-statement-due-to-statement-timeout.yml @@ -0,0 +1,5 @@ +--- +title: Stop merge requests with thousands of commits from timing out +merge_request: 15063 +author: +type: performance