2016-10-21 16:13:41 +00:00
|
|
|
# This configuration file should be exclusively used to set queue settings for
|
|
|
|
|
# Sidekiq. Any other setting should be specified using the Sidekiq CLI or the
|
|
|
|
|
# Sidekiq Ruby API (see config/initializers/sidekiq.rb).
|
|
|
|
|
---
|
|
|
|
|
# All the queues to process and their weights. Every queue _must_ have a weight
|
|
|
|
|
# defined.
|
|
|
|
|
#
|
|
|
|
|
# The available weights are as follows
|
|
|
|
|
#
|
|
|
|
|
# 1: low priority
|
|
|
|
|
# 2: medium priority
|
|
|
|
|
# 3: high priority
|
|
|
|
|
# 5: _super_ high priority, this should only be used for _very_ important queues
|
|
|
|
|
#
|
|
|
|
|
# As per http://stackoverflow.com/a/21241357/290102 the formula for calculating
|
|
|
|
|
# the likelihood of a job being popped off a queue (given all queues have work
|
|
|
|
|
# to perform) is:
|
|
|
|
|
#
|
|
|
|
|
# chance = (queue weight / total weight of all queues) * 100
|
|
|
|
|
:queues:
|
|
|
|
|
- [post_receive, 5]
|
|
|
|
|
- [merge, 5]
|
|
|
|
|
- [update_merge_requests, 3]
|
2017-02-17 11:38:22 +00:00
|
|
|
- [process_commit, 3]
|
2016-10-21 16:13:41 +00:00
|
|
|
- [new_note, 2]
|
2017-08-01 12:38:45 +00:00
|
|
|
- [new_issue, 2]
|
|
|
|
|
- [new_merge_request, 2]
|
2017-08-21 12:25:58 +00:00
|
|
|
- [pipeline_processing, 5]
|
2017-11-17 16:24:40 +00:00
|
|
|
- [pipeline_creation, 4]
|
2017-08-21 12:25:58 +00:00
|
|
|
- [pipeline_default, 3]
|
|
|
|
|
- [pipeline_cache, 3]
|
2018-11-05 00:37:40 +00:00
|
|
|
- [deployment, 3]
|
2019-05-27 12:21:36 +00:00
|
|
|
- [auto_merge, 3]
|
2017-08-21 12:02:20 +00:00
|
|
|
- [pipeline_hooks, 2]
|
2016-10-21 16:13:41 +00:00
|
|
|
- [gitlab_shell, 2]
|
|
|
|
|
- [email_receiver, 2]
|
|
|
|
|
- [emails_on_push, 2]
|
2016-10-22 05:28:39 +00:00
|
|
|
- [mailers, 2]
|
2018-03-30 13:05:20 +00:00
|
|
|
- [mail_scheduler, 2]
|
2017-06-22 12:18:01 +00:00
|
|
|
- [invalid_gpg_signature_update, 2]
|
2017-07-10 11:19:50 +00:00
|
|
|
- [create_gpg_signature, 2]
|
2017-11-28 16:16:50 +00:00
|
|
|
- [rebase, 2]
|
2017-02-22 19:25:06 +00:00
|
|
|
- [upload_checksum, 1]
|
2016-10-21 16:13:41 +00:00
|
|
|
- [repository_fork, 1]
|
|
|
|
|
- [repository_import, 1]
|
2017-10-13 16:50:36 +00:00
|
|
|
- [github_importer, 1]
|
2017-11-28 16:16:50 +00:00
|
|
|
- [github_import_advance_stage, 1]
|
2016-10-21 16:13:41 +00:00
|
|
|
- [project_service, 1]
|
|
|
|
|
- [delete_user, 1]
|
2018-07-16 18:30:17 +00:00
|
|
|
- [todos_destroyer, 1]
|
2016-09-21 14:15:12 +00:00
|
|
|
- [delete_merged_branches, 1]
|
2016-10-11 12:25:17 +00:00
|
|
|
- [authorized_projects, 1]
|
2016-10-21 16:13:41 +00:00
|
|
|
- [expire_build_instance_artifacts, 1]
|
|
|
|
|
- [group_destroy, 1]
|
|
|
|
|
- [irker, 1]
|
2017-05-08 15:13:02 +00:00
|
|
|
- [namespaceless_project_destroy, 1]
|
2016-10-21 16:13:41 +00:00
|
|
|
- [project_cache, 1]
|
|
|
|
|
- [project_destroy, 1]
|
|
|
|
|
- [project_export, 1]
|
2017-04-27 10:08:57 +00:00
|
|
|
- [web_hook, 1]
|
2016-10-21 16:13:41 +00:00
|
|
|
- [repository_check, 1]
|
|
|
|
|
- [git_garbage_collect, 1]
|
2016-12-17 04:09:50 +00:00
|
|
|
- [reactive_caching, 1]
|
2016-10-21 16:13:41 +00:00
|
|
|
- [cronjob, 1]
|
|
|
|
|
- [default, 1]
|
2015-12-15 21:48:23 +00:00
|
|
|
- [pages, 1]
|
2017-03-07 12:59:20 +00:00
|
|
|
- [system_hook_push, 1]
|
2017-05-05 16:01:33 +00:00
|
|
|
- [propagate_service_template, 1]
|
2017-06-02 15:12:36 +00:00
|
|
|
- [background_migration, 1]
|
2017-10-03 22:20:47 +00:00
|
|
|
- [gcp_cluster, 1]
|
2017-09-06 05:16:26 +00:00
|
|
|
- [project_migrate_hashed_storage, 1]
|
2019-01-17 01:57:35 +00:00
|
|
|
- [project_rollback_hashed_storage, 1]
|
2019-01-17 01:53:50 +00:00
|
|
|
- [hashed_storage, 1]
|
2018-02-06 13:25:46 +00:00
|
|
|
- [pages_domain_verification, 1]
|
2019-06-24 20:35:12 +00:00
|
|
|
- [pages_domain_ssl_renewal, 1]
|
2018-02-22 21:08:35 +00:00
|
|
|
- [object_storage_upload, 1]
|
2018-02-21 16:43:21 +00:00
|
|
|
- [object_storage, 1]
|
2018-02-26 12:15:51 +00:00
|
|
|
- [plugin, 1]
|
2018-03-06 11:20:36 +00:00
|
|
|
- [pipeline_background, 1]
|
2018-05-03 12:55:14 +00:00
|
|
|
- [repository_update_remote_mirror, 1]
|
|
|
|
|
- [repository_remove_remote, 1]
|
2018-05-16 15:46:18 +00:00
|
|
|
- [create_note_diff_file, 1]
|
2018-06-11 20:45:16 +00:00
|
|
|
- [delete_diff_files, 1]
|
Add repository languages for projects
Our friends at GitHub show the programming languages for a long time,
and inspired by that this commit means to create about the same
functionality.
Language detection is done through Linguist, as before, where the
difference is that we cache the result in the database. Also, Gitaly can
incrementaly scan a repository. This is done through a shell out, which
creates overhead of about 3s each run. For now this won't be improved.
Scans are triggered by pushed to the default branch, usually `master`.
However, one exception to this rule the charts page. If we're requesting
this expensive data anyway, we just cache it in the database.
Edge cases where there is no repository, or its empty are caught in the
Repository model. This makes use of Redis caching, which is probably
already loaded.
The added model is called RepositoryLanguage, which will make it harder
if/when GitLab supports multiple repositories per project. However, for
now I think this shouldn't be a concern. Also, Language could be
confused with the i18n languages and felt like the current name was
suiteable too.
Design of the Project#Show page is done with help from @dimitrieh. This
change is not visible to the end user unless detections are done.
2018-06-06 11:10:59 +00:00
|
|
|
- [detect_repository_languages, 1]
|
2018-09-06 19:20:42 +00:00
|
|
|
- [auto_devops, 2]
|
2019-01-10 14:22:58 +00:00
|
|
|
- [container_repository, 1]
|
Allow public forks to be deduplicated
When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.
Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.
When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.
These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.
Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.
To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.
Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)
In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.
2018-12-03 13:49:58 +00:00
|
|
|
- [object_pool, 1]
|
2018-11-19 15:03:58 +00:00
|
|
|
- [repository_cleanup, 1]
|
2018-10-07 18:31:08 +00:00
|
|
|
- [delete_stored_files, 1]
|
2018-12-05 15:22:52 +00:00
|
|
|
- [remote_mirror_notification, 2]
|
2018-12-13 19:26:56 +00:00
|
|
|
- [project_daily_statistics, 1]
|
2018-12-06 00:51:30 +00:00
|
|
|
- [import_issues_csv, 2]
|
2019-02-20 21:29:48 +00:00
|
|
|
- [chat_notification, 2]
|
2019-04-01 06:11:08 +00:00
|
|
|
- [migrate_external_diffs, 1]
|
|
|
|
|
- [update_project_statistics, 1]
|
2019-04-23 14:27:01 +00:00
|
|
|
- [phabricator_import_import_tasks, 1]
|
2019-07-02 14:44:39 +00:00
|
|
|
- [update_namespace_statistics, 1]
|
2019-07-16 20:10:44 +00:00
|
|
|
- [chaos, 2]
|
2019-06-13 12:19:12 +00:00
|
|
|
|
|
|
|
|
# EE-specific queues
|
|
|
|
|
- [ldap_group_sync, 2]
|
|
|
|
|
- [create_github_webhook, 2]
|
|
|
|
|
- [geo, 1]
|
|
|
|
|
- [repository_update_mirror, 1]
|
2019-08-27 21:50:17 +00:00
|
|
|
- [repository_push_audit_event, 1]
|
2019-06-13 12:19:12 +00:00
|
|
|
- [new_epic, 2]
|
|
|
|
|
- [project_import_schedule, 1]
|
|
|
|
|
- [project_update_repository_storage, 1]
|
|
|
|
|
- [admin_emails, 1]
|
|
|
|
|
- [elastic_batch_project_indexer, 1]
|
|
|
|
|
- [elastic_indexer, 1]
|
|
|
|
|
- [elastic_full_index, 1]
|
|
|
|
|
- [elastic_commit_indexer, 1]
|
|
|
|
|
- [elastic_namespace_indexer, 1]
|
|
|
|
|
- [export_csv, 1]
|
|
|
|
|
- [incident_management, 2]
|
2019-08-19 19:12:36 +00:00
|
|
|
- [jira_connect, 1]
|
2019-08-09 09:40:45 +00:00
|
|
|
- [update_external_pull_requests, 3]
|
2019-09-09 23:52:54 +00:00
|
|
|
- [refresh_license_compliance_checks, 2]
|
2019-09-23 00:06:29 +00:00
|
|
|
- [design_management_new_version, 1]
|