This refactors repository caching so it's possible to selectively
refresh certain caches, instead of just expiring and refreshing
everything.
To allow this the various methods that were cached (e.g. "tag_count" and
"readme") use a similar pattern that makes expiring and refreshing
their data much easier.
In this new setup caches are refreshed as follows:
1. After a commit (but before running ProjectCacheWorker) we expire some
basic caches such as the commit count and repository size.
2. ProjectCacheWorker will recalculate the commit count, repository
size, then refresh a specific set of caches based on the list of
files changed in a push payload.
This requires a bunch of changes to the various methods that may be
cached. For one, data should not be cached if a branch used or the
entire repository does not exist. To prevent all these methods from
handling this manually this is taken care of in
Repository#cache_method_output. Some methods still manually check for
the existence of a repository but this result is also cached.
With selective flushing implemented ProjectCacheWorker no longer uses an
exclusive lease for all of its work. Instead this worker only uses a
lease to limit the number of times the repository size is updated as
this is a fairly expensive operation.
When we updated gitlab_git to 10.4.1, `tag.target` changed from pointing
to the sha of the tag to the sha of the commit the tag points to. The
problem is that only annotated tags have `object_sha`s, lightweight tags
don't (it's nil), so (only) in their case we still need to use
`tag.target`.
When pushing commits to existing branches we don’t
need to flush branch git data (branch names / counts)
When flushes the cache when pushing commits skip to
flush branch and tag git data (names / counts) because
those operations are managed explicitly in each case
Repopulated expired cache as soon as possible
Rémy Coutable