gitlab-org--gitlab-foss/spec/workers
Yorick Peterse ffb9b3ef18
Refactor cache refreshing/expiring
This refactors repository caching so it's possible to selectively
refresh certain caches, instead of just expiring and refreshing
everything.

To allow this the various methods that were cached (e.g. "tag_count" and
"readme") use a similar pattern that makes expiring and refreshing
their data much easier.

In this new setup caches are refreshed as follows:

1. After a commit (but before running ProjectCacheWorker) we expire some
   basic caches such as the commit count and repository size.

2. ProjectCacheWorker will recalculate the commit count, repository
   size, then refresh a specific set of caches based on the list of
   files changed in a push payload.

This requires a bunch of changes to the various methods that may be
cached. For one, data should not be cached if a branch used or the
entire repository does not exist. To prevent all these methods from
handling this manually this is taken care of in
Repository#cache_method_output. Some methods still manually check for
the existence of a repository but this result is also cached.

With selective flushing implemented ProjectCacheWorker no longer uses an
exclusive lease for all of its work. Instead this worker only uses a
lease to limit the number of times the repository size is updated as
this is a fairly expensive operation.
2016-11-21 15:05:13 +01:00
..
concerns Re-organize queues to use for Sidekiq 2016-10-21 18:17:07 +02:00
repository_check Project tools visibility level 2016-09-01 11:47:59 -03:00
authorized_projects_worker_spec.rb Precalculate user's authorized projects in database 2016-11-18 20:25:45 +02:00
build_coverage_worker_spec.rb Update code coverage for CI build asynchronously 2016-10-13 15:22:55 +02:00
build_email_worker_spec.rb Update the other sites for reset_delivered_emails! as well 2016-10-17 07:25:20 +00:00
build_finished_worker_spec.rb Add build finished worker that creates a workflow 2016-10-14 14:45:14 +02:00
build_hooks_worker_spec.rb Perform CI build hooks asynchronously using worker 2016-10-13 12:45:16 +02:00
build_success_worker_spec.rb Do not process build success if project was removed 2016-10-13 14:56:32 +02:00
delete_merged_branches_worker_spec.rb Add button to delete all merged branches 2016-11-09 21:04:03 +01:00
delete_user_worker_spec.rb
email_receiver_worker_spec.rb
emails_on_push_worker_spec.rb Update the other sites for reset_delivered_emails! as well 2016-10-17 07:25:20 +00:00
every_sidekiq_worker_spec.rb Re-organize queues to use for Sidekiq 2016-10-21 18:17:07 +02:00
expire_build_artifacts_worker_spec.rb ExpireBuildArtifactsWorker query builds table without ordering enqueuing one job per build to cleanup 2016-10-07 15:33:58 +02:00
expire_build_instance_artifacts_worker_spec.rb Avoid race condition when expiring artifacts 2016-10-17 19:47:03 +02:00
git_garbage_collect_worker_spec.rb Remove unused 'require' 2016-11-04 14:34:10 +01:00
group_destroy_worker_spec.rb Fix bug where destroying a namespace would not always destroy projects 2016-08-11 15:36:35 -07:00
merge_worker_spec.rb Unify check branch name exist 2016-06-20 21:44:21 +02:00
new_note_worker_spec.rb Does not raise error when Note not found when processing NewNoteWorker 2016-11-11 22:54:11 -02:00
pipeline_hooks_worker_spec.rb Execute pipeline hooks asynchronously 2016-10-12 20:20:01 +02:00
pipeline_metrics_worker_spec.rb Added test events specs and logic. Also fixed some SQL and refactored the pipeline worker spec. 2016-11-17 08:22:55 +01:00
pipeline_notification_worker_spec.rb Send only to users have :read_build access, feedback: 2016-10-21 18:18:03 +08:00
pipeline_proccess_worker_spec.rb Rename pipeline workers to match current convention 2016-10-06 14:52:00 +02:00
pipeline_success_worker_spec.rb Process MWBS in successful pipeline asynchronously 2016-10-07 09:32:29 +02:00
pipeline_update_worker_spec.rb Rename pipeline workers to match current convention 2016-10-06 14:52:00 +02:00
post_receive_spec.rb Extract project#update_merge_requests and SystemHooks to its own worker from GitPushService 2016-10-13 17:23:24 +02:00
process_commit_worker_spec.rb Process commits in a separate worker 2016-11-07 13:11:44 +01:00
project_cache_worker_spec.rb Refactor cache refreshing/expiring 2016-11-21 15:05:13 +01:00
project_destroy_worker_spec.rb Fix skip_repo parameter being ignored when destroying a namespace 2016-08-03 17:07:38 -07:00
prune_old_events_worker_spec.rb Added cron to prune events older than 12 months. 2016-09-07 19:41:25 +02:00
remove_expired_group_links_worker_spec.rb Allow project group links to be expired 2016-08-18 22:45:41 +01:00
remove_expired_members_worker_spec.rb Allow project group links to be expired 2016-08-18 22:45:41 +01:00
remove_unreferenced_lfs_objects_worker_spec.rb Remove unreferenced LFS objects from DB and fs 2016-10-28 19:39:20 +02:00
repository_fork_worker_spec.rb Fix a bug where forking a project from a repository storage to another would fail 2016-07-26 17:22:13 -04:00
repository_import_worker_spec.rb
stuck_ci_builds_worker_spec.rb
trending_projects_worker_spec.rb Precalculate trending projects 2016-10-10 12:27:08 +02:00
update_merge_requests_worker_spec.rb Extract project#update_merge_requests and SystemHooks to its own worker from GitPushService 2016-10-13 17:23:24 +02:00