Commit graph

22 commits

Author SHA1 Message Date
GitLab Bot
6b75320f52 Add latest changes from gitlab-org/gitlab@master 2019-10-17 12:07:33 +00:00
GitLab Bot
8e45d25f7d Add latest changes from gitlab-org/gitlab@master 2019-10-16 18:08:01 +00:00
GitLab Bot
0dfbcd8f8b Add latest changes from gitlab-org/gitlab@master 2019-10-11 18:06:15 +00:00
GitLab Bot
9a1c545674 Add latest changes from gitlab-org/gitlab@master 2019-10-03 15:07:07 +00:00
GitLab Bot
927cfbfe63 Add latest changes from gitlab-org/gitlab@master 2019-10-03 12:06:00 +00:00
GitLab Bot
bd860c22f6 Add latest changes from gitlab-org/gitlab@master 2019-09-17 12:06:48 +00:00
Stan Hu
c34240d26f Log errors for failed pipeline creation in PostReceive
When a pipeline fails to create in `PostReceive`, the error is silently
discarded, making it difficult to understand why a pipeline was not
created. We now add a Sidekiq warning message for this. Adding a Sentry
exception when this happens would generate a lot of noise for invalid CI
files.

Relates to https://gitlab.com/gitlab-org/gitlab-ee/issues/14720
2019-09-05 21:42:14 -07:00
Stan Hu
b46b9d5e89 Fix pipelines not always being created after a push
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31741 introduced
a regression where not all the right parameters would be passed into
`Ci::CreatePipelineService`. We fix this by breaking out the pipeline
parameters and reusing a method from `Gitlab::DataBuilder::Push`.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/66196
2019-08-17 06:23:26 -07:00
Stan Hu
971e040c00 Merge branch 'dm-process-commit-worker-n+1' into 'master'
Look up upstream commits once before queuing ProcessCommitWorkers

Closes #65464

See merge request gitlab-org/gitlab-ce!31871
2019-08-16 20:26:38 +00:00
Stan Hu
f14647fdae Expire project caches once per push instead of once per ref
Previously `ProjectCacheWorker` would be scheduled once per ref, which
would generate unnecessary I/O and load on Sidekiq, especially if many
tags or branches were pushed at once. `ProjectCacheWorker` would expire
three items:

1. Repository size: This only needs to be updated once per push.
2. Commit count: This only needs to be updated if the default branch
   is updated.
3. Project method caches: This only needs to be updated if the default
   branch changes, but only if certain files change (e.g. README,
   CHANGELOG, etc.).

Because the third item requires looking at the actual changes in the
commit deltas, we schedule one `ProjectCacheWorker` to handle the first
two cases, and schedule a separate `ProjectCacheWorker` for the third
case if it is needed. As a result, this brings down the number of
`ProjectCacheWorker` jobs from N to 2.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/52046
2019-08-16 19:53:56 +00:00
Douwe Maan
97c2564ffa
Look up upstream commits once before queuing ProcessCommitWorkers
Instead of checking if a commit already exists in the upstream project
in its ProcessCommitWorker and bailing out if it does, we check the
existence of all commits in bulk in Git::BranchHooksService, so that we
can skip scheduling ProcessCommitWorker jobs for those commits
that already exist upstream entirely.
2019-08-16 13:31:48 -05:00
Nick Thomas
3cd40c4a80 Merge branch 'sh-optimize-commit-deltas-post-receive' into 'master'
Reduce Gitaly calls in PostReceive

Closes #65878

See merge request gitlab-org/gitlab-ce!31741
2019-08-14 14:38:28 +00:00
Stan Hu
e658f9603c Only expire tag cache once per push
Previously each tag in a push would invoke the Gitaly `FindAllTags` RPC
since the tag cache would be invalidated with every tag.

We can eliminate those extraneous calls by expiring the tag cache once
in `PostReceive` and taking advantage of the cached tags.

Relates to https://gitlab.com/gitlab-org/gitlab-ce/issues/65795
2019-08-13 09:04:30 -07:00
Stan Hu
4e2bb4e5e7 Reduce Gitaly calls in PostReceive
This commit reduces I/O load and memory utilization during PostReceive
for the common case when no project hooks or services are set up.

We saw a Gitaly N+1 issue in `CommitDelta` when many tags or branches
are pushed. We can reduce this overhead in the common case because we
observe that most new projects do not have any Web hooks or services,
especially when they are first created. Previously, `BaseHooksService`
unconditionally iterated through the last 20 commits of each ref to
build the `push_data` structure. The `push_data` structured was used in
numerous places:

1. Building the push payload in `EventCreateService`
2. Creating a CI pipeline
3. Executing project Web or system hooks
4. Executing project services
5. As the return value of `BaseHooksService#execute`
6. `BranchHooksService#invalidated_file_types`

We only need to generate the full `push_data` for items 3, 4, and 6.

Item 1: `EventCreateService` only needs the last commit and doesn't
actually need the commit deltas.

Item 2: In addition, `Ci::CreatePipelineService` only needed a subset of
the parameters.

Item 5: The return value of `BaseHooksService#execute` also wasn't being
used anywhere.

Item 6: This is only used when pushing to the default branch, so if
many tags are pushed we can save significant I/O here.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/65878

Fic
2019-08-12 22:28:49 -07:00
Nick Thomas
b2c73fde79
Look for new branches more carefully
In certain cases, GitLab can miss a PostReceive invocation the first
time a branch is pushed. When this happens, the "branch created" hooks
are not run, which means various features don't work until the branch
is deleted and pushed again.

This MR changes the `Git::BranchPushService` so it checks the cache of
existing branches in addition to the `oldrev` reported for the branch.
If the branch name isn't in the cache, chances are we haven't run the
service yet (it's what refreshes the cache), so we can go ahead and
run it, even through `oldrev` is set.

If the cache has been cleared by some other means in the meantime, then
we'll still fail to run the hooks when we should. Fixing that in the
general case is a larger problem, and we'd need to devote significant
engineering effort to it.

There's a chance that we'll run the relevant hooks *multiple times*
with this change, if there's a race between the branch being created,
and the `PostReceive` worker being run multiple times, but this can
already happen, since Sidekiq is "at-least-once" execution of jobs. So,
this should be safe.
2019-06-18 12:53:53 +01:00
Fabio Papa
352b43e0af Modify the branch hooks spec to expect processing of commit messages
Commit messages are not processed for references to issues when
creating the default branch on push. This was expected
behavior (probably to avoid performance problems when first pushing a
repository with thousands of commits). However, this is not an issue
because we always limit the number of commits to process to 100
regardless of whether we are creating the default branch or not.
2019-06-13 10:46:47 +00:00
Stan Hu
6c6e4ca495 Fix remote mirrors not updating after tag push
Remote mirrors were only being updated after pushes to branches, not
tags. This change consolidates the functionality into
Git::BaseHooksService so that both tags and branches will now update
remote mirrors.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/51240
2019-05-17 14:45:41 -07:00
Thong Kuah
85b29c1c2f Add frozen_string_literal to spec/services
Probably useful as we often move these files to "new" files.
2019-04-12 10:14:54 +12:00
Nick Thomas
e46d4bf4da
Extract a Git::{Base,Tag,Branch}HooksService 2019-04-05 18:26:53 +01:00
Nick Thomas
d17cce6aed
Only execute system hooks once when pushing tags 2019-04-03 01:53:38 +01:00
Nick Thomas
83fc1464dd
Rename GitPushService -> Git::BranchPushService 2019-03-25 13:02:31 +00:00
Nick Thomas
558e946263
Rename GitTagPushService -> Git::TagPushService 2019-03-25 12:56:46 +00:00