Commit graph

16 commits

Author SHA1 Message Date
Bob Van Landuyt
3598e60bf2 Add a Circuitbreaker for storage paths 2017-08-04 15:38:48 +02:00
Robert Speicher
72a7b30c9f Change all :empty_project to :project 2017-08-02 17:47:31 -04:00
Robert Speicher
9513bd18c4 Ensure all project factories use :repository trait or :empty_project 2017-08-01 14:51:52 -04:00
Lin Jen-Shin
f69c0f801f Remove unneeded asserts and add tests for inactive RequestStore 2017-07-18 18:57:56 +08:00
Lin Jen-Shin
3922b80329 Rename the methods to make it fit with current name 2017-07-18 18:04:20 +08:00
Lin Jen-Shin
ffc5b29bd0 Follow feedback on the merge request 2017-07-18 17:48:48 +08:00
Lin Jen-Shin
aada5273fa Use RequestStoreWrap for Commit#author
We also try to use instance variable to cache the result if
RequestStore is not available, so we could keep the same logic,
using the same cache key. Also introduce a way to specify method
specific cache key
2017-07-18 01:18:20 +08:00
Lin Jen-Shin
143fc48aba Add RequestStoreWrap to cache via RequestStore
I don't like the idea of `RequestStore` at all, because it's just a
global state which shouldn't be used at all. But we have a number of
places calling `ProtectedBranch.protected?` and `ProtectedTag.protected?`
in a loop for the same user, project, and ref whenever we're checking
against if the jobs for a given pipeline is accessible for a given user.
This means we're effectively making N queries for the same thing over
and over.

To properly fix this, we need to change how we check the permission,
and that could be a huge work. To solve this quickly, adding a cache
layer for the given request would be quite simple to do.

We're already doing this in Commit#author, and this is extending that
idea and make it generalized.
2017-07-17 23:24:46 +08:00
Paul Charlton
cb3b4a15e6 Support multiple Redis instances based on queue type 2017-07-11 03:35:47 +00:00
Grzegorz Bizon
0430b76441 Enable Style/DotPosition Rubocop 👮 2017-06-21 13:48:12 +00:00
Yorick Peterse
ac382b5682
Use CTEs for nested groups and authorizations
This commit introduces the usage of Common Table Expressions (CTEs) to
efficiently retrieve nested group hierarchies, without having to rely on
the "routes" table (which is an _incredibly_ inefficient way of getting
the data). This requires a patch to ActiveRecord (found in the added
initializer) to work properly as ActiveRecord doesn't support WITH
statements properly out of the box.

Unfortunately MySQL provides no efficient way of getting nested groups.
For example, the old routes setup could easily take 5-10 seconds
depending on the amount of "routes" in a database. Providing vastly
different logic for both MySQL and PostgreSQL will negatively impact the
development process. Because of this the various nested groups related
methods return empty relations when used in combination with MySQL.

For project authorizations the logic is split up into two classes:

* Gitlab::ProjectAuthorizations::WithNestedGroups
* Gitlab::ProjectAuthorizations::WithoutNestedGroups

Both classes get the fresh project authorizations (= as they should be
in the "project_authorizations" table), including nested groups if
PostgreSQL is used. The logic of these two classes is quite different
apart from their public interface. This complicates development a bit,
but unfortunately there is no way around this.

This commit also introduces Gitlab::GroupHierarchy. This class can be
used to get the ancestors and descendants of a base relation, or both by
using a UNION. This in turn is used by methods such as:

* Namespace#ancestors
* Namespace#descendants
* User#all_expanded_groups

Again this class relies on CTEs and thus only works on PostgreSQL. The
Namespace methods will return an empty relation when MySQL is used,
while User#all_expanded_groups will return only the groups a user is a
direct member of.

Performance wise the impact is quite large. For example, on GitLab.com
Namespace#descendants used to take around 580 ms to retrieve data for a
particular user. Using CTEs we are able to reduce this down to roughly 1
millisecond, returning the exact same data.

== On The Fly Refreshing

Refreshing of authorizations on the fly (= when
users.authorized_projects_populated was not set) is removed with this
commit. This simplifies the code, and ensures any queries used for
authorizations are not mutated because they are executed in a Rails
scope (e.g. Project.visible_to_user).

This commit includes a migration to schedule refreshing authorizations
for all users, ensuring all of them have their authorizations in place.
Said migration schedules users in batches of 5000, with 5 minutes
between every batch to smear the load around a bit.

== Spec Changes

This commit also introduces some changes to various specs. For example,
some specs for ProjectTeam assumed that creating a personal project
would _not_ lead to the owner having access, which is incorrect. Because
we also no longer refresh authorizations on the fly for new users some
code had to be added to the "empty_project" factory. This chunk of code
ensures that the owner's permissions are refreshed after creating the
project, something that is normally done in Projects::CreateService.
2017-05-17 16:51:08 +02:00
Bob Van Landuyt
019b06b9d2 Load a project's CI status in batch from redis 2017-04-26 12:04:22 +00:00
Bob Van Landuyt
516a405eb2 Take the ref of a pipeline into account when caching status 2017-04-07 17:24:11 +02:00
Bob Van Landuyt
02072e17ab Rename ProjectBuildStatus -> ProjectPipelineStatus 2017-04-07 17:24:11 +02:00
Bob Van Landuyt
a6d313001a Wrap updating of cache after pipeline transition in class method 2017-04-07 17:24:11 +02:00
Bob Van Landuyt
9082d1e046 Rename Ci::PipelineStatus -> Ci::ProjectBuildStatus 2017-04-07 17:24:11 +02:00