2020-07-16 18:09:35 +00:00
---
stage: Create
group: Gitaly
2020-11-26 06:09:20 +00:00
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-07-16 18:09:35 +00:00
type: reference, howto
---
2021-01-28 06:08:59 +00:00
# Repository storage types **(FREE SELF)**
2017-09-06 05:16:26 +00:00
2021-02-23 06:11:16 +00:00
GitLab can be configured to use one or multiple repository storages. These storages can be:
2019-03-11 16:38:19 +00:00
2021-02-23 06:11:16 +00:00
- Accessed via [Gitaly ](gitaly/index.md ), optionally on
2021-03-10 12:09:14 +00:00
[its own server ](gitaly/configure_gitaly.md#run-gitaly-on-its-own-server ).
2021-02-23 06:11:16 +00:00
- Mounted to the local disk. This [method ](repository_storage_paths.md#configure-repository-storage-paths )
is deprecated and [scheduled to be removed ](https://gitlab.com/groups/gitlab-org/-/epics/2320 ) in
GitLab 14.0.
- Exposed as an NFS shared volume. This method is deprecated and
[scheduled to be removed ](https://gitlab.com/groups/gitlab-org/-/epics/3371 ) in GitLab 14.0.
2019-03-11 16:38:19 +00:00
2021-02-23 06:11:16 +00:00
In GitLab:
2019-03-11 16:38:19 +00:00
2021-02-23 06:11:16 +00:00
- Repository storages are configured in:
- `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` configuration hash for Omnibus GitLab
installations.
- `gitlab.yml` by the `repositories.storages` key for installations from source.
- The `default` repository storage is available in any installations that haven't customized it. By
default, it points to a Gitaly node.
2019-03-11 16:38:19 +00:00
2021-02-23 06:11:16 +00:00
The repository storage types documented here apply to any repository storage defined in
`git_data_dirs({})` or `repositories.storages` .
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
## Hashed storage
2017-09-06 05:16:26 +00:00
2021-02-23 06:11:16 +00:00
> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28283) in GitLab 10.0.
> - Made the default for new installations in GitLab 12.0.
> - Enabled by default for new and renamed projects in GitLab 13.0.
Hashed storage stores projects on disk in a location based on a hash of the project's ID. Hashed
storage is different to [legacy storage ](#legacy-storage ) where a project is stored based on:
- The project's URL.
- The folder structure where the repository is stored on disk.
This makes the folder structure immutable and eliminates the need to synchronize state from URLs to
disk structure. This means that renaming a group, user, or project:
- Costs only the database transaction.
- Takes effect immediately.
The hash also helps spread the repositories more evenly on the disk. The top-level directory
contains fewer folders than the total number of top-level namespaces.
The hash format is based on the hexadecimal representation of a SHA256, calculated with
`SHA256(project.id)` . The top-level folder uses the first two characters, followed by another folder
with the next two characters. They are both stored in a special `@hashed` folder so they can
co-exist with existing legacy storage projects. For example:
2017-09-06 05:16:26 +00:00
```ruby
# Project's repository:
"@hashed/#{hash[0..1]}/#{hash[2..3]}/#{hash}.git"
# Wiki's repository:
"@hashed/#{hash[0..1]}/#{hash[2..3]}/#{hash}.wiki.git"
```
2021-02-23 06:11:16 +00:00
### Translate hashed storage paths
2020-03-20 09:09:22 +00:00
2021-02-23 06:11:16 +00:00
Troubleshooting problems with the Git repositories, adding hooks, and other tasks requires you
translate between the human-readable project name and the hashed storage path. You can translate:
- From a [project's name to its hashed path ](#from-project-name-to-hashed-path ).
- From a [hashed path to a project's name ](#from-hashed-path-to-project-name ).
2020-03-20 09:09:22 +00:00
#### From project name to hashed path
2021-02-23 06:11:16 +00:00
Administrators can look up a project's hashed path from its name or ID using:
- The [Admin area ](../user/admin_area/index.md#administering-projects ).
- A Rails console.
2020-03-20 09:09:22 +00:00
2021-02-23 06:11:16 +00:00
To look up a project's hash path in the Admin Area:
2020-03-20 09:09:22 +00:00
2021-06-15 18:09:57 +00:00
1. On the top bar, select **Menu >** ** {admin}** **Admin** .
1. On the left sidebar, select **Overview > Projects** and select the project.
2021-02-23 06:11:16 +00:00
The **Gitaly relative path** is displayed there and looks similar to:
2020-03-20 09:09:22 +00:00
2020-03-25 06:07:58 +00:00
```plaintext
2020-03-20 09:09:22 +00:00
"@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git"
```
2021-02-23 06:11:16 +00:00
To look up a project's hash path using a Rails console:
2020-03-20 09:09:22 +00:00
2021-02-23 06:11:16 +00:00
1. Start a [Rails console ](operations/rails_console.md#starting-a-rails-console-session ).
1. Run a command similar to this example (use either the project's ID or its name):
2020-03-20 09:09:22 +00:00
2021-02-23 06:11:16 +00:00
```ruby
Project.find(16).disk_path
Project.find_by_full_path('group/project').disk_path
```
2020-03-20 09:09:22 +00:00
#### From hashed path to project name
2021-02-23 06:11:16 +00:00
Administrators can look up a project's name from its hashed storage path using a Rails console. To
look up a project's name from its hashed storage path:
2020-03-20 09:09:22 +00:00
2020-10-14 15:08:42 +00:00
1. Start a [Rails console ](operations/rails_console.md#starting-a-rails-console-session ).
2021-02-23 06:11:16 +00:00
1. Run a command similar to this example:
2020-03-20 09:09:22 +00:00
2021-02-23 06:11:16 +00:00
```ruby
ProjectRepository.find_by(disk_path: '@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9').project
```
2020-03-20 09:09:22 +00:00
2021-02-23 06:11:16 +00:00
The quoted string in that command is the directory tree you can find on your GitLab server. For
example, on a default Omnibus installation this would be `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git`
2020-03-20 09:09:22 +00:00
with `.git` from the end of the directory name removed.
2021-02-23 06:11:16 +00:00
The output includes the project ID and the project name. For example:
2020-03-20 09:09:22 +00:00
2020-03-25 06:07:58 +00:00
```plaintext
2020-03-20 09:09:22 +00:00
=> #< Project id:16 it / supportteam / ticketsystem >
```
2019-03-11 16:38:19 +00:00
### Hashed object pools
Allow public forks to be deduplicated
When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.
Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.
When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.
These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.
Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.
To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.
Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)
In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.
2018-12-03 13:49:58 +00:00
2020-05-21 06:08:25 +00:00
> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1606) in GitLab 12.1.
2019-03-20 14:01:54 +00:00
2021-02-25 06:10:51 +00:00
Object pools are repositories used to deduplicate forks of public and internal projects and
contain the objects from the source project. Using `objects/info/alternates` , the source project and
forks use the object pool for shared objects. For more information, see
[How Git object deduplication works in GitLab ](../development/git_object_deduplication.md ).
2020-05-19 12:08:21 +00:00
2021-02-25 06:10:51 +00:00
Objects are moved from the source project to the object pool when housekeeping is run on the source
2021-04-30 00:10:28 +00:00
project. Object pool repositories are stored similarly to regular repositories in a directory called `@pools` instead of `@hashed`
Allow public forks to be deduplicated
When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.
Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.
When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.
These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.
Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.
To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.
Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)
In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.
2018-12-03 13:49:58 +00:00
```ruby
# object pool paths
"@pools/#{hash[0..1]}/#{hash[2..3]}/#{hash}.git"
```
2021-02-25 06:10:51 +00:00
WARNING:
2021-04-30 00:10:28 +00:00
Do not run `git prune` or `git gc` in object pool repositories, which are stored in the `@pools` directory.
This can cause data loss in the regular repositories that depend on the object pool.
2021-02-25 06:10:51 +00:00
### Object storage support
This table shows which storable objects are storable in each storage type:
| Storable object | Legacy storage | Hashed storage | S3 compatible | GitLab version |
|:-----------------|:---------------|:---------------|:--------------|:---------------|
| Repository | Yes | Yes | - | 10.0 |
| Attachments | Yes | Yes | - | 10.2 |
| Avatars | Yes | No | - | - |
| Pages | Yes | No | - | - |
| Docker Registry | Yes | No | - | - |
| CI/CD job logs | No | No | - | - |
| CI/CD artifacts | No | No | Yes | 9.4 / 10.6 |
| CI/CD cache | No | No | Yes | - |
| LFS objects | Yes | Similar | Yes | 10.0 / 10.7 |
| Repository pools | No | Yes | - | 11.6 |
Files stored in an S3-compatible endpoint can have the same advantages as
[hashed storage ](#hashed-storage ), as long as they are not prefixed with
`#{namespace}/#{project_name}` . This is true for CI/CD cache and LFS objects.
2018-06-27 03:01:09 +00:00
2020-05-19 12:08:21 +00:00
#### Avatars
2018-06-27 03:01:09 +00:00
2021-02-25 06:10:51 +00:00
Each file is stored in a directory that matches the `id` assigned to it in the database. The
filename is always `avatar.png` for user avatars. When an avatar is replaced, the `Upload` model is
destroyed and a new one takes place with a different `id` .
#### CI/CD artifacts
2018-06-27 03:01:09 +00:00
2021-02-25 06:10:51 +00:00
CI/CD artifacts are:
2018-06-27 03:01:09 +00:00
2021-02-25 06:10:51 +00:00
- S3-compatible since GitLab 9.4, initially available in [GitLab Premium ](https://about.gitlab.com/pricing/ ).
- Available in [GitLab Free ](https://about.gitlab.com/pricing/ ) since GitLab 10.6.
2018-06-27 03:01:09 +00:00
2020-05-19 12:08:21 +00:00
#### LFS objects
2018-06-27 03:01:09 +00:00
2020-04-01 18:07:56 +00:00
[LFS Objects in GitLab ](../topics/git/lfs/index.md ) implement a similar
2021-02-25 06:10:51 +00:00
storage pattern using two characters and two-level folders, following Git's own implementation:
2018-06-27 03:01:09 +00:00
```ruby
"shared/lfs-objects/#{oid[0..1}/#{oid[2..3]}/#{oid[4..-1]}"
# Based on object `oid`: `8909029eb962194cfb326259411b22ae3f4a814b5be4f80651735aeef9f3229c`, path will be:
"shared/lfs-objects/89/09/029eb962194cfb326259411b22ae3f4a814b5be4f80651735aeef9f3229c"
```
2021-02-25 06:10:51 +00:00
LFS objects are also [S3-compatible ](lfs/index.md#storing-lfs-objects-in-remote-object-storage ).
2020-05-19 12:08:21 +00:00
## Legacy storage
2020-12-08 00:09:45 +00:00
WARNING:
2021-02-25 06:10:51 +00:00
In GitLab 13.0, legacy storage is deprecated. If you haven't migrated to hashed storage yet, check
the [migration instructions ](raketasks/storage.md#migrate-to-hashed-storage ). Support for legacy
storage is [scheduled to be removed ](https://gitlab.com/gitlab-org/gitaly/-/issues/1690 ) in GitLab
14.0. In GitLab 13.0 and later, switching new projects to legacy storage is not possible. The
option to choose between hashed and legacy storage in the Admin Area is disabled.
Legacy storage was the storage behavior prior to version GitLab 10.0. For historical reasons,
GitLab replicated the same mapping structure from the projects URLs:
- Project's repository: `#{namespace}/#{project_name}.git` .
- Project's wiki: `#{namespace}/#{project_name}.wiki.git` .
This structure enabled you to migrate from existing solutions to GitLab, and for Administrators to
find where the repository was stored. This approach also had some drawbacks:
- Storage location concentrated a large number of top-level namespaces. The impact could be
reduced by [multiple repository storage paths ](repository_storage_paths.md ).
- Because backups were a snapshot of the same URL mapping, if you tried to recover a very old
backup, you needed to verify whether any project had taken the place of an old removed or renamed
project sharing the same URL. This meant that `mygroup/myproject` from your backup may not have
been the same original project that was at that same URL today.
- Any change in the URL needed to be reflected on disk (when groups, users, or projects were
renamed. This could add a lot of load in big installations, especially if using any type of
network-based file system.