896c0bdbfb
When a project is forked, the new repository used to be a deep copy of everything stored on disk by leveraging `git clone`. This works well, and makes isolation between repository easy. However, the clone is at the start 100% the same as the origin repository. And in the case of the objects in the object directory, this is almost always going to be a lot of duplication. Object Pools are a way to create a third repository that essentially only exists for its 'objects' subdirectory. This third repository's object directory will be set as alternate location for objects. This means that in the case an object is missing in the local repository, git will look in another location. This other location is the object pool repository. When Git performs garbage collection, it's smart enough to check the alternate location. When objects are duplicated, it will allow git to throw one copy away. This copy is on the local repository, where to pool remains as is. These pools have an origin location, which for now will always be a repository that itself is not a fork. When the root of a fork network is forked by a user, the fork still clones the full repository. Async, the pool repository will be created. Either one of these processes can be done earlier than the other. To handle this race condition, the Join ObjectPool operation is idempotent. Given its idempotent, we can schedule it twice, with the same effect. To accommodate the holding of state two migrations have been added. 1. Added a state column to the pool_repositories column. This column is managed by the state machine, allowing for hooks on transitions. 2. pool_repositories now has a source_project_id. This column in convenient to have for multiple reasons: it has a unique index allowing the database to handle race conditions when creating a new record. Also, it's nice to know who the host is. As that's a short link to the fork networks root. Object pools are only available for public project, which use hashed storage and when forking from the root of the fork network. (That is, the project being forked from itself isn't a fork) In this commit message I use both ObjectPool and Pool repositories, which are alike, but different from each other. ObjectPool refers to whatever is on the disk stored and managed by Gitaly. PoolRepository is the record in the database. |
||
---|---|---|
.. | ||
environments | ||
initializers | ||
locales | ||
prometheus | ||
routes | ||
application.rb | ||
boot.rb | ||
database.yml.env | ||
database.yml.mysql | ||
database.yml.postgresql | ||
dependency_decisions.yml | ||
environment.rb | ||
gitlab.yml.example | ||
jest.config.js | ||
karma.config.js | ||
license_finder.yml | ||
mail_room.yml | ||
no_todos_messages.yml | ||
object_store_settings.rb | ||
puma.example.development.rb | ||
README.md | ||
redis.cache.yml.example | ||
redis.queues.yml.example | ||
redis.shared_state.yml.example | ||
resque.yml.example | ||
routes.rb | ||
secrets.yml.example | ||
settings.rb | ||
sidekiq.yml.example | ||
sidekiq_queues.yml | ||
spring.rb | ||
unicorn.rb.example | ||
unicorn.rb.example.development | ||
webpack.config.js |
Configuration files Documentation
Note that most configuration files (config/*.*
) committed into
gitlab-ce will not be used for
omnibus-gitlab. Configuration
files committed into gitlab-ce are only used for development.
gitlab.yml
You can find most of GitLab configuration settings here.
mail_room.yml
This file is actually an YML wrapped inside an ERB file to enable templated
values to be specified from gitlab.yml
. mail_room loads this file first as
an ERB file and then loads the resulting YML as its configuration.
resque.yml
This file is called resque.yml
for historical reasons. We are NOT
using Resque at the moment. It is used to specify Redis configuration
values when a single database instance of Redis is desired.
Advanced Redis configuration files
In more advanced configurations of Redis key-value storage, it is desirable to separate the keys by lifecycle and intended use to ease provisioning and management of scalable Redis clusters.
These settings provide routing and other configuration data (such as sentinel, persistence policies, and other Redis customization) for connections to Redis single instances, Redis sentinel, and Redis clusters.
If desired, the routing URL provided by these settings can be used with:
- Unix Socket
- named socket for each Redis instance desired.
database number
for each Redis instance desired.
- TCP Socket
host name
or IP for each Redis instance desired- TCP port number for each Redis instance desired
database number
for each Redis instance desired
Example URL attribute formats for GitLab Redis .yml
configuration files
- Unix Socket, default Redis database (0)
url: unix:/path/to/redis.sock
url: unix:/path/to/redis.sock?db=
- Unix Socket, Redis database 44
url: unix:/path/to/redis.sock?db=44
url: unix:/path/to/redis.sock?extra=foo&db=44
- TCP Socket for Redis on localhost, port 6379, database 33
url: redis://:mynewpassword@localhost:6379/33
- TCP Socket for Redis on remote host
myserver
, port 6379, database 33url: redis://:mynewpassword@myserver:6379/33
redis.cache.yml
If configured, redis.cache.yml
overrides the
resque.yml
settings to configure the Redis database instance
used for Rails.cache
and other volatile non-persistent data which enhances
the performance of GitLab.
Settings here can be overridden by the environment variable
GITLAB_REDIS_CACHE_CONFIG_FILE
which provides
an alternate location for configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for cache
is:
- URL from a configuration file pointed to by the
GITLAB_REDIS_CACHE_CONFIG_FILE
environment variable - URL from
redis.cache.yml
- URL from a configuration file pointed to by the
GITLAB_REDIS_CONFIG_FILE
environment variable - URL from
resque.yml
redis://localhost:6380
The order of precedence for all other configuration settings for cache
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
- the configuration file pointed to by the
GITLAB_REDIS_CACHE_CONFIG_FILE
environment variable - the configuration file
redis.cache.yml
- the configuration file pointed to by the
GITLAB_REDIS_CONFIG_FILE
environment variable - the configuration file
resque.yml
redis.queues.yml
If configured, redis.queues.yml
overrides the
resque.yml
settings to configure the Redis database instance
used for clients of ::Gitlab::Redis::Queues
.
These queues are intended to be the foundation
of reliable inter-process communication between modules, whether on the same
host node, or within a cluster. The primary clients of the queues are
SideKiq, Mailroom, CI Runner, Workhorse, and push services. Settings here can
be overridden by the environment variable
GITLAB_REDIS_QUEUES_CONFIG_FILE
which provides an alternate location for
configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for queues
is:
- URL from a configuration file pointed to by the
GITLAB_REDIS_QUEUES_CONFIG_FILE
environment variable - URL from
redis.queues.yml
- URL from a configuration file pointed to by the
GITLAB_REDIS_CONFIG_FILE
environment variable - URL from
resque.yml
redis://localhost:6381
The order of precedence for all other configuration settings for queues
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
- the configuration file pointed to by the
GITLAB_REDIS_QUEUES_CONFIG_FILE
environment variable - the configuration file
redis.queues.yml
- the configuration file pointed to by the
GITLAB_REDIS_CONFIG_FILE
environment variable - the configuration file
resque.yml
redis.shared_state.yml
If configured, redis.shared_state.yml
overrides the
resque.yml
settings to configure the Redis database instance
used for clients of ::Gitlab::Redis::SharedState
such as session state,
and rate limiting.
Settings here can be overridden by the environment variable
GITLAB_REDIS_SHARED_STATE_CONFIG_FILE
which provides
an alternate location for configuration settings.
The order of precedence for the URL used to connect to the Redis instance
used for shared_state
is:
- URL from a configuration file pointed to by the
GITLAB_REDIS_SHARED_STATE_CONFIG_FILE
environment variable - URL from
redis.shared_state.yml
- URL from a configuration file pointed to by the
GITLAB_REDIS_CONFIG_FILE
environment variable - URL from
resque.yml
redis://localhost:6382
The order of precedence for all other configuration settings for shared_state
are selected from only the first of the following files found (if a setting
is not provided in an earlier file, the remainder of the files are not
searched):
- the configuration file pointed to by the
GITLAB_REDIS_SHARED_STATE_CONFIG_FILE
environment variable - the configuration file
redis.shared_state.yml
- the configuration file pointed to by the
GITLAB_REDIS_CONFIG_FILE
environment variable - the configuration file
resque.yml