gitlab-org--gitlab-foss/spec/workers
Zeger-Jan van de Weg 896c0bdbfb
Allow public forks to be deduplicated
When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.

Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.

When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.

These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.

Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.

To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.

Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)

In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.
2018-12-07 19:18:37 +01:00
..
auto_devops Automatically disable Auto DevOps for project upon first pipeline failure 2018-09-06 19:20:42 +00:00
ci Improve comments 2018-11-29 15:13:36 +09:00
concerns Removes all the irrelevant import related code and columns 2018-11-27 12:58:13 +00:00
deployments Squashed commit of the following: 2018-11-06 10:54:27 +09:00
gitlab/github_import Removes all the irrelevant import related code and columns 2018-11-27 12:58:13 +00:00
mail_scheduler
object_pool Allow public forks to be deduplicated 2018-12-07 19:18:37 +01:00
repository_check Remove broken git storage for testing 2018-10-23 10:36:47 +02:00
todos_destroyer
admin_email_worker_spec.rb
archive_trace_worker_spec.rb Improve spec 2018-11-29 14:51:03 +09:00
authorized_projects_worker_spec.rb
background_migration_worker_spec.rb
build_coverage_worker_spec.rb
build_finished_worker_spec.rb
build_hooks_worker_spec.rb
build_success_worker_spec.rb Create deployments only if action: start 2018-11-06 10:54:27 +09:00
build_trace_sections_worker_spec.rb
cluster_platform_configure_worker_spec.rb Create k8s namespace for project in group clusters 2018-12-05 10:16:44 +13:00
cluster_provision_worker_spec.rb Incorporates Kubernetes Namespace into Cluster's flow 2018-11-02 15:46:15 +00:00
cluster_wait_for_ingress_ip_address_worker_spec.rb
create_gpg_signature_worker_spec.rb
create_note_diff_file_worker_spec.rb
create_pipeline_worker_spec.rb
delete_container_repository_worker_spec.rb Delete a container registry asynchronously 2018-09-07 12:42:59 -07:00
delete_diff_files_worker_spec.rb
delete_merged_branches_worker_spec.rb
delete_user_worker_spec.rb
detect_repository_languages_worker_spec.rb
email_receiver_worker_spec.rb Apply patches when creating MR via email 2018-11-07 16:27:55 +01:00
emails_on_push_worker_spec.rb Fix typos in lib 2018-11-15 12:15:43 +02:00
every_sidekiq_worker_spec.rb Backport of gitlab-ee!8470 2018-11-19 16:29:19 -05:00
expire_build_artifacts_worker_spec.rb
expire_build_instance_artifacts_worker_spec.rb
expire_job_cache_worker_spec.rb
expire_pipeline_cache_worker_spec.rb
git_garbage_collect_worker_spec.rb Remove Gitlab::Git::Repository#rugged and Gollum code 2018-10-02 16:34:28 -03:00
gitlab_shell_worker_spec.rb
gitlab_usage_ping_worker_spec.rb
group_destroy_worker_spec.rb
invalid_gpg_signature_update_worker_spec.rb
issue_due_scheduler_worker_spec.rb
merge_worker_spec.rb
namespaceless_project_destroy_worker_spec.rb Remove the `ForkedProjectLink` model 2018-10-19 11:57:20 +02:00
new_issue_worker_spec.rb
new_merge_request_worker_spec.rb
new_note_worker_spec.rb
pages_domain_verification_cron_worker_spec.rb
pages_domain_verification_worker_spec.rb
pipeline_hooks_worker_spec.rb
pipeline_metrics_worker_spec.rb
pipeline_notification_worker_spec.rb
pipeline_process_worker_spec.rb
pipeline_schedule_worker_spec.rb Rename project's pipelines relation 2018-12-05 14:39:15 +00:00
pipeline_success_worker_spec.rb
pipeline_update_worker_spec.rb
plugin_worker_spec.rb
post_receive_spec.rb Removes idenfitication by commit from Gitlab::Identifier 2018-10-25 11:36:26 +01:00
process_commit_worker_spec.rb
project_cache_worker_spec.rb
project_destroy_worker_spec.rb
project_export_worker_spec.rb
project_migrate_hashed_storage_worker_spec.rb
project_service_worker_spec.rb Log project services errors when executing async 2018-09-10 18:01:25 -03:00
propagate_service_template_worker_spec.rb
prune_old_events_worker_spec.rb Added new test and updated all event ids 2018-10-05 17:12:31 +00:00
prune_web_hook_logs_worker_spec.rb Improve PruneWebHookLogsWorker spec 2018-12-07 10:55:44 +01:00
reactive_caching_worker_spec.rb
rebase_worker_spec.rb Fix deprecation: You are passing an instance of ActiveRecord::Base to find 2018-11-26 23:27:04 +01:00
remove_expired_group_links_worker_spec.rb
remove_expired_members_worker_spec.rb
remove_unreferenced_lfs_objects_worker_spec.rb
repository_cleanup_worker_spec.rb Use BFG object maps to clean projects 2018-12-06 18:58:00 +00:00
repository_fork_worker_spec.rb Remove the `ForkedProjectLink` model 2018-10-19 11:57:20 +02:00
repository_import_worker_spec.rb Removes all the irrelevant import related code and columns 2018-11-27 12:58:13 +00:00
repository_remove_remote_worker_spec.rb Remove Gitlab::Git::Repository#rugged and Gollum code 2018-10-02 16:34:28 -03:00
repository_update_remote_mirror_worker_spec.rb
run_pipeline_schedule_worker_spec.rb Revert "Avoid conflicts between enumlator's schedule! method and state machine's schedule! method" 2018-10-04 16:35:05 +09:00
stage_update_worker_spec.rb
storage_migrator_worker_spec.rb
stuck_ci_jobs_worker_spec.rb Lock writes to trace stream 2018-11-27 17:55:20 +01:00
stuck_import_jobs_worker_spec.rb StuckImportJobsWorker query performance optimization 2018-11-07 21:24:31 +00:00
stuck_merge_jobs_worker_spec.rb
system_hook_push_worker_spec.rb
trending_projects_worker_spec.rb
update_head_pipeline_for_merge_request_worker_spec.rb Merge request pipelines 2018-12-05 15:57:52 +09:00
update_merge_requests_worker_spec.rb
upload_checksum_worker_spec.rb
wait_for_cluster_creation_worker_spec.rb