gitlab-org--gitlab-foss/config
Zeger-Jan van de Weg 896c0bdbfb
Allow public forks to be deduplicated
When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.

Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.

When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.

These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.

Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.

To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.

Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)

In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.
2018-12-07 19:18:37 +01:00
..
environments Explicitly set locale fallbacks 2018-11-21 12:17:26 +01:00
initializers Merge branch 'dm-remove-prune-web-hook-logs-worker' into 'master' 2018-12-07 16:11:44 +00:00
locales Resolve "Create new group: Rename form fields and update UI" 2018-10-30 16:23:47 +00:00
prometheus Fix common_metrics.yml 2018-09-06 11:43:12 +02:00
routes Merge branch '19376-post-bfg-cleanup' into 'master' 2018-12-06 20:43:58 +00:00
application.rb Merge branch 'security-182-update-workhorse' into 'master' 2018-11-28 19:06:30 -05:00
boot.rb Eliminate duplicated words 2018-11-22 01:01:23 +09:00
database.yml.env
database.yml.mysql
database.yml.postgresql
dependency_decisions.yml Adds a reference to the echarts licence file 2018-12-06 12:31:00 +00:00
environment.rb Switch to Rails 5 by default 2018-11-14 12:38:30 +01:00
gitlab.yml.example Support RSA and ECDSA algorithms in Omniauth JWT 2018-12-05 18:17:40 +01:00
jest.config.js Setup Jest test environment 2018-12-05 09:24:42 +01:00
karma.config.js Make DefinePlugin definitions more specific 2018-10-02 16:24:44 -05:00
license_finder.yml
mail_room.yml
no_todos_messages.yml
object_store_settings.rb Make ObjectStoreSettings use more explicit and add specs 2018-07-24 14:44:44 +03:00
puma.example.development.rb Add experimental support for Puma 2018-10-25 17:50:15 +01:00
README.md
redis.cache.yml.example
redis.queues.yml.example
redis.shared_state.yml.example
resque.yml.example
routes.rb Merge branch '52767-more-chaos-for-gitlab' into 'master' 2018-11-07 16:21:17 +00:00
secrets.yml.example
settings.rb Ensure that db encryption keys have proper bytesize 2018-11-22 15:35:49 +01:00
sidekiq.yml.example
sidekiq_queues.yml Allow public forks to be deduplicated 2018-12-07 19:18:37 +01:00
spring.rb [Rails5] Update files by rails app:update 2018-03-22 09:37:57 +11:00
unicorn.rb.example Add experimental support for Puma 2018-10-25 17:50:15 +01:00
unicorn.rb.example.development Add experimental support for Puma 2018-10-25 17:50:15 +01:00
webpack.config.js Suggests issues when typing title 2018-11-27 15:10:40 +00:00

Configuration files Documentation

Note that most configuration files (config/*.*) committed into gitlab-ce will not be used for omnibus-gitlab. Configuration files committed into gitlab-ce are only used for development.

gitlab.yml

You can find most of GitLab configuration settings here.

mail_room.yml

This file is actually an YML wrapped inside an ERB file to enable templated values to be specified from gitlab.yml. mail_room loads this file first as an ERB file and then loads the resulting YML as its configuration.

resque.yml

This file is called resque.yml for historical reasons. We are NOT using Resque at the moment. It is used to specify Redis configuration values when a single database instance of Redis is desired.

Advanced Redis configuration files

In more advanced configurations of Redis key-value storage, it is desirable to separate the keys by lifecycle and intended use to ease provisioning and management of scalable Redis clusters.

These settings provide routing and other configuration data (such as sentinel, persistence policies, and other Redis customization) for connections to Redis single instances, Redis sentinel, and Redis clusters.

If desired, the routing URL provided by these settings can be used with:

  1. Unix Socket
    1. named socket for each Redis instance desired.
    2. database number for each Redis instance desired.
  2. TCP Socket
    1. host name or IP for each Redis instance desired
    2. TCP port number for each Redis instance desired
    3. database number for each Redis instance desired

Example URL attribute formats for GitLab Redis .yml configuration files

  • Unix Socket, default Redis database (0)
    • url: unix:/path/to/redis.sock
    • url: unix:/path/to/redis.sock?db=
  • Unix Socket, Redis database 44
    • url: unix:/path/to/redis.sock?db=44
    • url: unix:/path/to/redis.sock?extra=foo&db=44
  • TCP Socket for Redis on localhost, port 6379, database 33
    • url: redis://:mynewpassword@localhost:6379/33
  • TCP Socket for Redis on remote host myserver, port 6379, database 33
    • url: redis://:mynewpassword@myserver:6379/33

redis.cache.yml

If configured, redis.cache.yml overrides the resque.yml settings to configure the Redis database instance used for Rails.cache and other volatile non-persistent data which enhances the performance of GitLab. Settings here can be overridden by the environment variable GITLAB_REDIS_CACHE_CONFIG_FILE which provides an alternate location for configuration settings.

The order of precedence for the URL used to connect to the Redis instance used for cache is:

  1. URL from a configuration file pointed to by the GITLAB_REDIS_CACHE_CONFIG_FILE environment variable
  2. URL from redis.cache.yml
  3. URL from a configuration file pointed to by the GITLAB_REDIS_CONFIG_FILE environment variable
  4. URL from resque.yml
  5. redis://localhost:6380

The order of precedence for all other configuration settings for cache are selected from only the first of the following files found (if a setting is not provided in an earlier file, the remainder of the files are not searched):

  1. the configuration file pointed to by the GITLAB_REDIS_CACHE_CONFIG_FILE environment variable
  2. the configuration file redis.cache.yml
  3. the configuration file pointed to by the GITLAB_REDIS_CONFIG_FILE environment variable
  4. the configuration file resque.yml

redis.queues.yml

If configured, redis.queues.yml overrides the resque.yml settings to configure the Redis database instance used for clients of ::Gitlab::Redis::Queues. These queues are intended to be the foundation of reliable inter-process communication between modules, whether on the same host node, or within a cluster. The primary clients of the queues are SideKiq, Mailroom, CI Runner, Workhorse, and push services. Settings here can be overridden by the environment variable GITLAB_REDIS_QUEUES_CONFIG_FILE which provides an alternate location for configuration settings.

The order of precedence for the URL used to connect to the Redis instance used for queues is:

  1. URL from a configuration file pointed to by the GITLAB_REDIS_QUEUES_CONFIG_FILE environment variable
  2. URL from redis.queues.yml
  3. URL from a configuration file pointed to by the GITLAB_REDIS_CONFIG_FILE environment variable
  4. URL from resque.yml
  5. redis://localhost:6381

The order of precedence for all other configuration settings for queues are selected from only the first of the following files found (if a setting is not provided in an earlier file, the remainder of the files are not searched):

  1. the configuration file pointed to by the GITLAB_REDIS_QUEUES_CONFIG_FILE environment variable
  2. the configuration file redis.queues.yml
  3. the configuration file pointed to by the GITLAB_REDIS_CONFIG_FILE environment variable
  4. the configuration file resque.yml

redis.shared_state.yml

If configured, redis.shared_state.yml overrides the resque.yml settings to configure the Redis database instance used for clients of ::Gitlab::Redis::SharedState such as session state, and rate limiting. Settings here can be overridden by the environment variable GITLAB_REDIS_SHARED_STATE_CONFIG_FILE which provides an alternate location for configuration settings.

The order of precedence for the URL used to connect to the Redis instance used for shared_state is:

  1. URL from a configuration file pointed to by the GITLAB_REDIS_SHARED_STATE_CONFIG_FILE environment variable
  2. URL from redis.shared_state.yml
  3. URL from a configuration file pointed to by the GITLAB_REDIS_CONFIG_FILE environment variable
  4. URL from resque.yml
  5. redis://localhost:6382

The order of precedence for all other configuration settings for shared_state are selected from only the first of the following files found (if a setting is not provided in an earlier file, the remainder of the files are not searched):

  1. the configuration file pointed to by the GITLAB_REDIS_SHARED_STATE_CONFIG_FILE environment variable
  2. the configuration file redis.shared_state.yml
  3. the configuration file pointed to by the GITLAB_REDIS_CONFIG_FILE environment variable
  4. the configuration file resque.yml