gitlab-org--gitlab-foss/config/sidekiq_queues.yml

115 lines
3.2 KiB
YAML
Raw Normal View History

# This configuration file should be exclusively used to set queue settings for
# Sidekiq. Any other setting should be specified using the Sidekiq CLI or the
# Sidekiq Ruby API (see config/initializers/sidekiq.rb).
---
# All the queues to process and their weights. Every queue _must_ have a weight
# defined.
#
# The available weights are as follows
#
# 1: low priority
# 2: medium priority
# 3: high priority
# 5: _super_ high priority, this should only be used for _very_ important queues
#
# As per http://stackoverflow.com/a/21241357/290102 the formula for calculating
# the likelihood of a job being popped off a queue (given all queues have work
# to perform) is:
#
# chance = (queue weight / total weight of all queues) * 100
:queues:
- [post_receive, 5]
- [merge, 5]
- [update_merge_requests, 3]
- [process_commit, 3]
- [new_note, 2]
- [new_issue, 2]
- [new_merge_request, 2]
- [pipeline_processing, 5]
- [pipeline_creation, 4]
- [pipeline_default, 3]
- [pipeline_cache, 3]
Squashed commit of the following: commit 931d6ab0e025b0268d94e455f736b09a025e0578 Merge: b34d165320d 93846eb152f Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Nov 5 09:36:58 2018 +0900 Merge branch 'master-ce' into stateful_deployments commit b34d165320d6f3298c8b776ba66270a59c217412 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 18:07:08 2018 +0900 Fix flaky spec commit b5e0527c5d4fe8f18b2fdda5916bae9b8cd859a4 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 15:32:03 2018 +0900 Fix spec commit f78a5e96e66fe2d25086df495e339b470a274df8 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 14:59:29 2018 +0900 Remove unnecessary line in schema.rb commit 6ce7c483e0591b5d6f9588a99853834327b80031 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 14:55:48 2018 +0900 Add partial index for filling deployment at migration commit aecccfb5118c8982db3ba502fdf37b5e639fbfc6 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 14:42:24 2018 +0900 Fix fill empty finished at migration commit 0199e1761ad1b391ae87a53a9a113d3256529e0e Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 14:19:44 2018 +0900 Fix flaky spec commit 56ac84cd8095afab5b909119445537b7da06a2ff Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Nov 2 10:06:49 2018 +0900 Fix guard clause to prevent multiple deployments to a job commit 521561b6b303b54635c30cb23d78e49d14cec53d Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 20:19:24 2018 +0900 Fix spec commit 2878da0d29b9bd2dde69a1b216203df118dd59a1 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 19:38:59 2018 +0900 Simplify the factory commit 22fd7df02133f3a21828554965fd5619905eac2c Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 19:33:50 2018 +0900 Simplify the Deployable and BuildSuccessWorker commit 41108959677ed614f4548443a2f4303c4c04925a Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 18:34:20 2018 +0900 Fix spec commit ae75fe7461ac72f621498797f478d42331342b84 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 17:19:12 2018 +0900 Fix weird virtual deployment status commit 380fee7494d06407dccc292c3cbedbcee7b6e235 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 15:59:31 2018 +0900 Fix spec commit 29889fcbaadb3bbfd2f11c10bfbf5dceb3e3ddba Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 15:07:10 2018 +0900 Fix coding offence commit 36ac13f345f5ef25725c2236a791a40a3a9e6126 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 14:22:17 2018 +0900 Squashed commit of the following: commit ba9aede922e1643db3f06c56736d46d6d86d356b Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 14:21:33 2018 +0900 Fix ambiguious factory specification in update deployment service spec commit 013afb5668cb30dc4ca5b21945c17b341e7ea7f9 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Nov 1 14:10:24 2018 +0900 Fix spec commit 78793670d049e2dfb5fc98177eb4d10f20b9310b Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 18:26:12 2018 +0900 Fix spec commit 73d27e87c66698f2e3a817bb8728f02475b7ba4f Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 16:22:14 2018 +0900 Fix index commit 8580a226ea68bf5e49b35bfb5f404968bbfaf8e9 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 15:34:57 2018 +0900 Fix deployment relationships in Ci::Build commit d6d28b55afd1179200b4f5188e0b53079ff3c1a7 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 15:27:53 2018 +0900 Fix spec commit 94eb754e2e1bb9a1fe627f86823f571a8298d27b Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 31 14:07:11 2018 +0900 Fix spec commit 0b30f80bcd08a7a06bdde3378ec1733f865284be Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 20:15:31 2018 +0900 Fix spec commit 466bdcdb6af8cdb475c9fa16bd7d1dff23b11e40 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 19:28:51 2018 +0900 Fix spec commit a7c3caac99139e70fe3f1f3d14856939fa25c527 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 17:33:47 2018 +0900 Fix factory commit cea28ae100532e6711ce1d22676719a94e2da8a0 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 16:28:18 2018 +0900 Drop leagacy success commit 3785d685eabc10b6597cf3db67bf08385ccf298a Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:37:28 2018 +0900 Remove unnecessary migration file commit 0d597fa46eeffdbb9a4afb53005a8183e433c6bf Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:35:53 2018 +0900 Fix schema.rb commit ec3c2abc6944e09f6410468ae5e356865ec7b02b Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:34:21 2018 +0900 Rename post migration file commit 0e7281885a84656acf95f0f423732680f8fec076 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:31:01 2018 +0900 Remove include EnumWithNil commit b3846d59c07e07275126c70361bde7f30810729e Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 15:05:50 2018 +0900 Decouple action commit c9f9ba4eae9ca1edc7d8751e1d2e0572cb222d9c Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 14:23:29 2018 +0900 Remove status mock commit d95bfea1ca67b3a27a3226a669c2b1266d696682 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 14:17:14 2018 +0900 Add action commit 0cec39e0f76c22a18498f46d65ad7226fb30c3f8 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 30 13:44:07 2018 +0900 Remove unnecessary line in schema.rb commit 7b4c5f8e1b00dd8e6aa944352f9d8a9f3ae6f1c7 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 19:59:41 2018 +0900 Revert build success worker commit 0c52ffa4a23eea488c187317e8b400369846f399 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 19:11:47 2018 +0900 Use add_column_with_default properly commit ba9bae357da5dfd2f6ec05f7f9db9d0b31224f48 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 18:40:55 2018 +0900 Fix with_status commit 75dffc97b9c5f6fa73d9d09b125c8f849fa2caae Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 16:26:56 2018 +0900 Remove unnecessary line in schema.rb commit 25188ccc52fb29ca63b9205c4d95ffc2e0afadee Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 29 16:26:17 2018 +0900 Set default values in regular migration commit 98ea037fbf39c8d9f0db77fb50e2d08382425158 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 17:27:49 2018 +0900 Fix static analysis commit e7d1765f77f9ff9b94a34985a7855bdaab1da675 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 16:37:10 2018 +0900 Remove empty spec commit 0033f521ed1eae8117dba231961aa47c068bbcfb Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 16:34:55 2018 +0900 Simplify spec changes commit 0be4c6b3ade6d9a8bf28bcd177c66ebd7bb7d20a Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 16:32:45 2018 +0900 Simplify spec changes commit a93d25d79df7e25bdf688fc938c712922f9dc4df Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 16:02:31 2018 +0900 Fix flaky spec commit 339ad50cf471ca706b29f008ccd2bb881dd5b776 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 15:06:22 2018 +0900 Rename Deployments Success worker commit bd69c78085adcb9b0f8ff9b7041ae355953ad7ab Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 26 14:43:03 2018 +0900 Fix coding offence commit 004748b2a9c5236ec13eb01289418f3d6571c92c Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 20:09:10 2018 +0900 Rename to update deployment service commit b04a85e761de501f030f3844fd485a2b9e46f7f7 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 18:46:52 2018 +0900 Add spec for Project commit 548af23a5a07f0c20b72849d03aa0b98a0b49134 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 18:43:25 2018 +0900 Fix spec commit c977e4d3f17194c46a1bf857b473017ce21ef7e9 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 17:58:07 2018 +0900 Add spec for Environment commit 73feb9010f8d8093bee4b46e56d30cfef3e8e34a Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 17:39:24 2018 +0900 Add spec for Deployment model commit 9a3cfbf766f402571588839375cf311bb9807035 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 17:18:02 2018 +0900 Fix statis analysis commit a30d28dbc631a29855883ca89c592a10c012f1d2 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 17:17:32 2018 +0900 Ignore nil instance commit fa6fdd89f380e588a6bcf14b1f9aef0d14d3854b Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 16:20:40 2018 +0900 Add spec for deployable concern commit aa91186821dc671df2c7a641e37586dd5dfc1008 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 15:37:23 2018 +0900 Clean up deployable commit 34d3e18731f7906a3db250b105a64d1db83c2fca Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 15:13:05 2018 +0900 Fix 17 cycle analytics commit 8dc9e00408f9b390175e7d5ea743eed4fb9e3f79 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 25 13:56:51 2018 +0900 Fix static analysys commit 5c4175807a537bafc4b889b0a97e8f96f0e483cd Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 24 15:05:05 2018 +0900 Skip unnecessary sidekiq worker commit 9d8b5d423f49cc247c96ce3767d03b4af305809f Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 24 14:53:13 2018 +0900 Add changelog commit c8cabba496722240cadf7c161c80bceb09727cba Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 24 14:44:45 2018 +0900 Squashed commit of the following: commit f7643885ac2329e18d690a4e4f2d7614b732c793 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 19:38:45 2018 +0900 Fix deployment widget specs commit 03bd04b5c98b634dff6a0ab4292c150a9031995c Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 17:50:16 2018 +0900 Fix env status spec commit 4a49c6502b161a12f0f62d5ec167dff777047dab Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 17:48:59 2018 +0900 Fix environment spec commit 4044822887987e20a703990ff20352a532eeb965 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 17:47:17 2018 +0900 Fix environment spec commit 9939d44b7eb9da371de74c0f04fed1eb3db37ad3 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 15:45:43 2018 +0900 Add a new spec for deployment success worker commit f61c4d3657b5ef13b5da171460da68a6643ad4b5 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 15:38:11 2018 +0900 Fix cycle analytics helper commit b6242615e8298fb7fc047c8df8006c25ad717c70 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 14:41:54 2018 +0900 Fix cycle analysis helper commit 9a001cb4c4ed6f3b87dc612bdffc60a6b2b0a132 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 14:37:08 2018 +0900 Ignore coding offence in build success worker's spec commit 1fb88583025bac8a56172cbd59be04258ea4c5f3 Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 23 14:33:11 2018 +0900 Added more spec for deployments commit 1a6ba97ababbf62e8dd0ae0c56d75ab1268fd0ce Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 19:36:50 2018 +0900 Move after create hookd into success worker commit 09de5fed5d6f108423779cf9d9e7f1d21f3c1c91 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 19:30:06 2018 +0900 Fix build spec commit 73a55cbcabbb1e928eca3e53e8ff75dec178bc90 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 19:08:43 2018 +0900 Fix update_deployment_metrics_service_spec.rb commit ee05136a02ae9fa348b4b89b9a69937ebb9697dd Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 17:32:05 2018 +0900 Remove unnecessary degelate commit e246ddeebc01a807ccc36fdb484c3e72ad91e680 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 16:07:39 2018 +0900 Remove unnecessary optimistic locking commit dcc225c8237b90e3bc8dcc3dc2e3252e0b0be093 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 16:00:22 2018 +0900 Simplify status replication commit 13a5fd7afb67ba2712fcaecaea5fedf05f9ad177 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 15:29:24 2018 +0900 Fix sidekiq queue names commit dcc796f48d523538e1c91b9cd3e1c7065e5329b1 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 15:23:55 2018 +0900 Revert success check in update_merge_request_metrics commit 129ef083d637d4acb8c97a6d9ab96deb2ff6efcd Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 15:18:31 2018 +0900 Fix queue name of deployment success worker commit 10fe5a6484f4f02322ce5bb16844fc7b1d565963 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 15:09:42 2018 +0900 Introduce deployable module commit d91260bbe105bf46f6c06d9e9593c8c4cd5139cf Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 14:05:31 2018 +0900 Add database index for successful deployments commit 74274147263de4b60870065a19935498ce662e30 Author: Shinya Maeda <shinya@gitlab.com> Date: Mon Oct 22 13:51:59 2018 +0900 Fix invalid state transition commit ff18463cc847bf3cf5a3e49f3651eedfdf67c7e6 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 20:05:15 2018 +0900 Fix coding style offence commit 0202c0f5b631601edab7b359b087b307f5eb7ba3 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 18:34:07 2018 +0900 Target only successful deployments from other relations commit 1f2758cb030dec1df5dda30f6bc3e25b6d0841c9 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 18:21:28 2018 +0900 Add namespace explicitly commit 3d9227b6e5642cecde88d4edac925125f6474b11 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 17:42:30 2018 +0900 Fix spec in DeleteInconsistentInternalIdRecords commit 3e0cc99ff6c5c7188511618228a6ec027752ce69 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 16:10:16 2018 +0900 Fixed spec commit 8de09b8bb31f7b9f24ecdf9f2dd8ef358a260263 Author: Shinya Maeda <shinya@gitlab.com> Date: Fri Oct 19 14:22:35 2018 +0900 Fix create deployment service commit 31957570b4444492eeb412e765f96a56416c25f3 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 20:21:26 2018 +0900 Move CreateDeploymentService. Fix Cycle analytics spec and fixture. commit d2eb433a1bb9710c0d4778c4f34c12b6b64f60e6 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 20:11:22 2018 +0900 Fix build success worker commit 25e6cd87138bcdb69de8785ca367e479c8dbcc59 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 19:49:13 2018 +0900 Fix create deployment service spec commit d268bf410bf65e86c81eb76d50aa8e145b32d249 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 19:01:23 2018 +0900 Fix cycle analysys spec's deployment commit 525ade8aa1e4394ed8a759bb0437e407fbe74a35 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 18:24:04 2018 +0900 Fix factory to set legacy status by default commit c6a990821ac0a1ffa49e20e2d78d94b8ce075914 Author: Shinya Maeda <shinya@gitlab.com> Date: Thu Oct 18 17:25:40 2018 +0900 Remove unnecessary lib from deployment commit a6107e0e85ac26ee09da3316ebc11de32f067d82 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 17:38:58 2018 +0900 Fix recursive call commit 15c5f3b64061a75af3c3039ca7f49b1cc4ff3068 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 17:30:44 2018 +0900 Add finished_at commit c8d3d70366f694d78acb7e30d342c7697798b922 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 15:55:31 2018 +0900 Fix last_deployment methods as it used to return successful deployment always commit 96bbe8670cece021766fde95fe573cbbe23d1e55 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 15:49:57 2018 +0900 Redefine statuses commit c86a9d0bd2ab3e7a00bf61f094a96ee99b76b289 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 14:50:27 2018 +0900 Fix schema.rb commit 9ff5f0eaafbc08795018c7bb282b19f6327dee21 Author: Shinya Maeda <shinya@gitlab.com> Date: Wed Oct 17 14:18:04 2018 +0900 Default status nil to success commit 5928bd9bb94e1e8908ed1561e01595be84d5f4ec Author: Shinya Maeda <shinya@gitlab.com> Date: Tue Oct 16 15:13:48 2018 +0900 Add status to Deployment
2018-11-04 19:37:40 -05:00
- [deployment, 3]
- [auto_merge, 3]
- [pipeline_hooks, 2]
- [gitlab_shell, 2]
- [email_receiver, 2]
- [emails_on_push, 2]
- [mailers, 2]
- [mail_scheduler, 2]
- [invalid_gpg_signature_update, 2]
2017-07-10 07:19:50 -04:00
- [create_gpg_signature, 2]
2017-11-28 11:16:50 -05:00
- [rebase, 2]
- [upload_checksum, 1]
- [repository_fork, 1]
- [repository_import, 1]
Rewrite the GitHub importer from scratch Prior to this MR there were two GitHub related importers: * Github::Import: the main importer used for GitHub projects * Gitlab::GithubImport: importer that's somewhat confusingly used for importing Gitea projects (apparently they have a compatible API) This MR renames the Gitea importer to Gitlab::LegacyGithubImport and introduces a new GitHub importer in the Gitlab::GithubImport namespace. This new GitHub importer uses Sidekiq for importing multiple resources in parallel, though it also has the ability to import data sequentially should this be necessary. The new code is spread across the following directories: * lib/gitlab/github_import: this directory contains most of the importer code such as the classes used for importing resources. * app/workers/gitlab/github_import: this directory contains the Sidekiq workers, most of which simply use the code from the directory above. * app/workers/concerns/gitlab/github_import: this directory provides a few modules that are included in every GitHub importer worker. == Stages The import work is divided into separate stages, with each stage importing a specific set of data. Stages will schedule the work that needs to be performed, followed by scheduling a job for the "AdvanceStageWorker" worker. This worker will periodically check if all work is completed and schedule the next stage if this is the case. If work is not yet completed this worker will reschedule itself. Using this approach we don't have to block threads by calling `sleep()`, as doing so for large projects could block the thread from doing any work for many hours. == Retrying Work Workers will reschedule themselves whenever necessary. For example, hitting the GitHub API's rate limit will result in jobs rescheduling themselves. These jobs are not processed until the rate limit has been reset. == User Lookups Part of the importing process involves looking up user details in the GitHub API so we can map them to GitLab users. The old importer used an in-memory cache, but this obviously doesn't work when the work is spread across different threads. The new importer uses a Redis cache and makes sure we only perform API/database calls if absolutely necessary. Frequently used keys are refreshed, and lookup misses are also cached; removing the need for performing API/database calls if we know we don't have the data we're looking for. == Performance & Models The new importer in various places uses raw INSERT statements (as generated by `Gitlab::Database.bulk_insert`) instead of using Rails models. This allows us to bypass any validations and callbacks, drastically reducing the number of SQL queries and Gitaly RPC calls necessary to import projects. To ensure the code produces valid data the corresponding tests check if the produced rows are valid according to the model validation rules.
2017-10-13 12:50:36 -04:00
- [github_importer, 1]
2017-11-28 11:16:50 -05:00
- [github_import_advance_stage, 1]
- [project_service, 1]
- [delete_user, 1]
- [todos_destroyer, 1]
- [delete_merged_branches, 1]
- [authorized_projects, 1]
- [expire_build_instance_artifacts, 1]
- [group_destroy, 1]
- [irker, 1]
- [namespaceless_project_destroy, 1]
- [project_cache, 1]
- [project_destroy, 1]
- [project_export, 1]
- [web_hook, 1]
- [repository_check, 1]
- [git_garbage_collect, 1]
- [reactive_caching, 1]
- [cronjob, 1]
- [default, 1]
2015-12-15 16:48:23 -05:00
- [pages, 1]
- [system_hook_push, 1]
2017-05-05 12:01:33 -04:00
- [propagate_service_template, 1]
- [background_migration, 1]
- [gcp_cluster, 1]
- [project_migrate_hashed_storage, 1]
- [project_rollback_hashed_storage, 1]
- [hashed_storage, 1]
- [pages_domain_verification, 1]
- [pages_domain_ssl_renewal, 1]
- [object_storage_upload, 1]
2018-02-21 11:43:21 -05:00
- [object_storage, 1]
- [plugin, 1]
- [pipeline_background, 1]
- [repository_update_remote_mirror, 1]
- [repository_remove_remote, 1]
- [create_note_diff_file, 1]
- [delete_diff_files, 1]
- [detect_repository_languages, 1]
- [auto_devops, 2]
- [container_repository, 1]
Allow public forks to be deduplicated When a project is forked, the new repository used to be a deep copy of everything stored on disk by leveraging `git clone`. This works well, and makes isolation between repository easy. However, the clone is at the start 100% the same as the origin repository. And in the case of the objects in the object directory, this is almost always going to be a lot of duplication. Object Pools are a way to create a third repository that essentially only exists for its 'objects' subdirectory. This third repository's object directory will be set as alternate location for objects. This means that in the case an object is missing in the local repository, git will look in another location. This other location is the object pool repository. When Git performs garbage collection, it's smart enough to check the alternate location. When objects are duplicated, it will allow git to throw one copy away. This copy is on the local repository, where to pool remains as is. These pools have an origin location, which for now will always be a repository that itself is not a fork. When the root of a fork network is forked by a user, the fork still clones the full repository. Async, the pool repository will be created. Either one of these processes can be done earlier than the other. To handle this race condition, the Join ObjectPool operation is idempotent. Given its idempotent, we can schedule it twice, with the same effect. To accommodate the holding of state two migrations have been added. 1. Added a state column to the pool_repositories column. This column is managed by the state machine, allowing for hooks on transitions. 2. pool_repositories now has a source_project_id. This column in convenient to have for multiple reasons: it has a unique index allowing the database to handle race conditions when creating a new record. Also, it's nice to know who the host is. As that's a short link to the fork networks root. Object pools are only available for public project, which use hashed storage and when forking from the root of the fork network. (That is, the project being forked from itself isn't a fork) In this commit message I use both ObjectPool and Pool repositories, which are alike, but different from each other. ObjectPool refers to whatever is on the disk stored and managed by Gitaly. PoolRepository is the record in the database.
2018-12-03 08:49:58 -05:00
- [object_pool, 1]
2018-11-19 10:03:58 -05:00
- [repository_cleanup, 1]
- [delete_stored_files, 1]
- [remote_mirror_notification, 2]
- [project_daily_statistics, 1]
- [import_issues_csv, 2]
- [chat_notification, 2]
- [migrate_external_diffs, 1]
- [update_project_statistics, 1]
- [phabricator_import_import_tasks, 1]
- [update_namespace_statistics, 1]
2019-06-13 08:19:12 -04:00
# EE-specific queues
- [ldap_group_sync, 2]
- [create_github_webhook, 2]
- [geo, 1]
- [repository_update_mirror, 1]
- [new_epic, 2]
- [project_import_schedule, 1]
- [project_update_repository_storage, 1]
- [admin_emails, 1]
- [elastic_batch_project_indexer, 1]
- [elastic_indexer, 1]
- [elastic_full_index, 1]
- [elastic_commit_indexer, 1]
- [elastic_namespace_indexer, 1]
- [export_csv, 1]
- [incident_management, 2]