Avoid hitting statement timeout finding MR pipelines

For MRs with many thousands of commits, `SELECT DISTINCT(sha)` will be very
slow.

What we can't do to fix this:

1. Add an index. Postgres won't use it for DISTINCT without a lot of ceremony.
2. Do the `uniq` in Ruby. That can still be very slow with hundreds of
   thousands of commits.
3. Use a subquery. We haven't removed the `st_commits` column yet, but we will
   soon.

Until 3 is available to us, we can just do 2, but also add a limit clause. There
is no ordering, so this may return different results, but our goal with these
MRs is just to get them to load, so it's not a huge deal.
This commit is contained in:
Sean McGivern 2017-10-27 14:20:55 +01:00
parent e9d3526123
commit ba9b4c4de8
2 changed files with 6 additions and 1 deletions

View File

@ -874,7 +874,7 @@ class MergeRequest < ActiveRecord::Base
#
def all_commit_shas
if persisted?
column_shas = MergeRequestDiffCommit.where(merge_request_diff: merge_request_diffs).pluck('DISTINCT(sha)')
column_shas = MergeRequestDiffCommit.where(merge_request_diff: merge_request_diffs).limit(10_000).pluck('sha')
serialised_shas = merge_request_diffs.where.not(st_commits: nil).flat_map(&:commit_shas)
(column_shas + serialised_shas).uniq

View File

@ -0,0 +1,5 @@
---
title: Stop merge requests with thousands of commits from timing out
merge_request: 15063
author:
type: performance