When a pipeline fails to create in `PostReceive`, the error is silently
discarded, making it difficult to understand why a pipeline was not
created. We now add a Sidekiq warning message for this. Adding a Sentry
exception when this happens would generate a lot of noise for invalid CI
files.
Relates to https://gitlab.com/gitlab-org/gitlab-ee/issues/14720
Previously `ProjectCacheWorker` would be scheduled once per ref, which
would generate unnecessary I/O and load on Sidekiq, especially if many
tags or branches were pushed at once. `ProjectCacheWorker` would expire
three items:
1. Repository size: This only needs to be updated once per push.
2. Commit count: This only needs to be updated if the default branch
is updated.
3. Project method caches: This only needs to be updated if the default
branch changes, but only if certain files change (e.g. README,
CHANGELOG, etc.).
Because the third item requires looking at the actual changes in the
commit deltas, we schedule one `ProjectCacheWorker` to handle the first
two cases, and schedule a separate `ProjectCacheWorker` for the third
case if it is needed. As a result, this brings down the number of
`ProjectCacheWorker` jobs from N to 2.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/52046
This commit reduces I/O load and memory utilization during PostReceive
for the common case when no project hooks or services are set up.
We saw a Gitaly N+1 issue in `CommitDelta` when many tags or branches
are pushed. We can reduce this overhead in the common case because we
observe that most new projects do not have any Web hooks or services,
especially when they are first created. Previously, `BaseHooksService`
unconditionally iterated through the last 20 commits of each ref to
build the `push_data` structure. The `push_data` structured was used in
numerous places:
1. Building the push payload in `EventCreateService`
2. Creating a CI pipeline
3. Executing project Web or system hooks
4. Executing project services
5. As the return value of `BaseHooksService#execute`
6. `BranchHooksService#invalidated_file_types`
We only need to generate the full `push_data` for items 3, 4, and 6.
Item 1: `EventCreateService` only needs the last commit and doesn't
actually need the commit deltas.
Item 2: In addition, `Ci::CreatePipelineService` only needed a subset of
the parameters.
Item 5: The return value of `BaseHooksService#execute` also wasn't being
used anywhere.
Item 6: This is only used when pushing to the default branch, so if
many tags are pushed we can save significant I/O here.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/65878
Fic
Remote mirrors were only being updated after pushes to branches, not
tags. This change consolidates the functionality into
Git::BaseHooksService so that both tags and branches will now update
remote mirrors.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/51240
This adds a limitation that we will try to create pipeline
for at most 4 first changes (branches and tags).
This does not affect processing of Pipelines for Merge Requests,
as each updated MR will have associated pipeline created.
Previously the raw push option Array was sent to Pipeline::Chain::Skip.
This commit updates this class (and the chain of classes that pass the
push option parameters from the API internal `post_receive` endpoint to
that class) to treat push options as a Hash of options parsed by
GitLab::PushOptions.
The GitLab::PushOptions class takes options like this:
-o ci.skip -o merge_request.create -o merge_request.target=branch
and turns them into a Hash like this:
{
ci: {
skip: true
},
merge_request: {
create: true,
target: 'branch'
}
}
This now how Pipeline::Chain::Skip is determining if the `ci.skip` push
option was used.
GitLab Bot