This commit introduces a Sidekiq worker that precalculates the list of
trending projects on a daily basis. The resulting set is stored in a
database table that is then queried by Project.trending.
This setup means that Unicorn workers no longer _may_ have to calculate
the list of trending projects. Furthermore it supports filtering without
any complex caching mechanisms.
The data in the "trending_projects" table is inserted in the same order
as the project ranking. This means that getting the projects in the
correct order is simply a matter of:
SELECT projects.*
FROM projects
INNER JOIN trending_projects ON trending_projects.project_id = projects.id
ORDER BY trending_projects.id ASC;
Such a query will only take a few milliseconds at most (as measured on
GitLab.com), opposed to a few seconds for the query used for calculating
the project ranks.
The migration in this commit does not require downtime and takes care of
populating an initial list of trending projects.
== Public Projects
This finder class now _only_ returns public projects. Previously this
finder would also return private and internal projects. Including these
projects makes caching data much harder and less efficient. Meanwhile
including this data isn't very useful as very few users would be
interested in seeing projects they have access to as trending. That is,
the feature is more useful when you want to see what _other_ popular
projects there are.
== Caching
The data returned by TrendingProjectsFinder is now cached for a day
based on the number of months the data should be restricted to. The
cache is not flushed explicitly, instead it's rebuilt whenever it
expires.
== Timings
To measure the impact I changed the finder code to use the last 24
months instead of the last month. I then executed and measured 10
requests to the explore page. On the current "master" branch (commit
88fa5916ff) this would take an average of
2.43 seconds. Using the changes of this commit this was reduced to
around 1.7 seconds.
Fixesgitlab-org/gitlab-ce#22164
- Split up SnippetsController into separate dashboard and explore sections.
- Use consistent page titles, header titles and sidebars between dashboard and explore sections when signed in or not.
Eric Eastwood