2020-10-24 00:08:35 +00:00
---
2020-12-01 18:09:42 +00:00
stage: Enablement
group: Distribution
2020-11-26 06:09:20 +00:00
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-10-24 00:08:35 +00:00
---
2021-01-28 06:08:59 +00:00
# Repository storage Rake tasks **(FREE SELF)**
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
This is a collection of Rake tasks to help you list and migrate
existing projects and their attachments to the new
[hashed storage ](../repository_storage_types.md ) that GitLab
uses to organize the Git data.
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
## List projects and attachments
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
The following Rake tasks will list the projects and attachments that are
available on legacy and hashed storage.
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
### On legacy storage
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
To have a summary and then a list of projects and their attachments using legacy storage:
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
- **Omnibus installation**
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
```shell
# Projects
sudo gitlab-rake gitlab:storage:legacy_projects
sudo gitlab-rake gitlab:storage:list_legacy_projects
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
# Attachments
sudo gitlab-rake gitlab:storage:legacy_attachments
sudo gitlab-rake gitlab:storage:list_legacy_attachments
```
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
- **Source installation**
2018-06-07 15:40:44 +00:00
2020-05-19 12:08:21 +00:00
```shell
# Projects
sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_projects RAILS_ENV=production
2018-06-07 15:40:44 +00:00
2020-05-19 12:08:21 +00:00
# Attachments
sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_attachments RAILS_ENV=production
```
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
### On hashed storage
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
To have a summary and then a list of projects and their attachments using hashed storage:
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
- **Omnibus installation**
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
```shell
# Projects
sudo gitlab-rake gitlab:storage:hashed_projects
sudo gitlab-rake gitlab:storage:list_hashed_projects
2019-05-05 15:21:25 +00:00
2020-05-19 12:08:21 +00:00
# Attachments
sudo gitlab-rake gitlab:storage:hashed_attachments
sudo gitlab-rake gitlab:storage:list_hashed_attachments
```
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
- **Source installation**
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
```shell
# Projects
sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_projects RAILS_ENV=production
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
# Attachments
sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_attachments RAILS_ENV=production
```
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
## Migrate to hashed storage
2019-03-11 16:38:19 +00:00
2020-12-08 00:09:45 +00:00
WARNING:
2020-05-19 12:08:21 +00:00
In GitLab 13.0, [hashed storage ](../repository_storage_types.md#hashed-storage )
is enabled by default and the legacy storage is deprecated.
2021-04-26 03:10:04 +00:00
GitLab 14.0 eliminates support for legacy storage. If you're on GitLab
2020-05-19 12:08:21 +00:00
13.0 and later, switching new projects to legacy storage is not possible.
The option to choose between hashed and legacy storage in the admin area has
been disabled.
2019-03-11 16:38:19 +00:00
2021-01-12 00:10:42 +00:00
This task must be run on any machine that has Rails/Sidekiq configured and will
schedule all your existing projects and attachments associated with it to be
2020-12-10 12:09:43 +00:00
migrated to the **Hashed** storage type:
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
- **Omnibus installation**
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
```shell
sudo gitlab-rake gitlab:storage:migrate_to_hashed
```
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
- **Source installation**
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
```shell
sudo -u git -H bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production
```
If you have any existing integration, you may want to do a small rollout first,
to validate. You can do so by specifying an ID range with the operation by using
the environment variables `ID_FROM` and `ID_TO` . For example, to limit the rollout
to project IDs 50 to 100 in an Omnibus GitLab installation:
2019-03-11 16:38:19 +00:00
2020-01-30 15:09:15 +00:00
```shell
2020-05-19 12:08:21 +00:00
sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100
2019-03-11 16:38:19 +00:00
```
2020-07-29 00:09:37 +00:00
You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page.
2020-05-19 12:08:21 +00:00
There is a specific queue you can watch to see how long it will take to finish:
`hashed_storage:hashed_storage_project_migrate` .
2019-03-11 16:38:19 +00:00
2020-08-31 21:10:19 +00:00
After it reaches zero, you can confirm every project has been migrated by running the commands above.
2020-05-19 12:08:21 +00:00
If you find it necessary, you can run this migration script again to schedule missing projects.
2019-03-11 16:38:19 +00:00
2021-04-26 03:10:04 +00:00
Any error or warning is logged in Sidekiq's log file.
2019-03-11 16:38:19 +00:00
2020-09-15 15:10:08 +00:00
If [Geo ](../geo/index.md ) is enabled, each project that is successfully migrated
2020-05-19 12:08:21 +00:00
generates an event to replicate the changes on any **secondary** nodes.
2017-11-18 07:12:37 +00:00
2021-04-26 21:10:25 +00:00
You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but there are
[additional commands(#list-projects-and-attachments) to help you inspect projects and attachments in both legacy and hashed storage.
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
## Rollback from hashed storage to legacy storage
2017-09-06 05:16:26 +00:00
2020-12-08 00:09:45 +00:00
WARNING:
2020-05-19 12:08:21 +00:00
In GitLab 13.0, [hashed storage ](../repository_storage_types.md#hashed-storage )
is enabled by default and the legacy storage is deprecated.
2021-04-26 03:10:04 +00:00
GitLab 14.0 eliminates support for legacy storage. If you're on GitLab
2020-05-19 12:08:21 +00:00
13.0 and later, switching new projects to legacy storage is not possible.
The option to choose between hashed and legacy storage in the admin area has
been disabled.
2017-09-06 05:16:26 +00:00
2021-04-26 03:10:04 +00:00
This task schedules all your existing projects and associated attachments to be rolled back to the
2020-05-19 12:08:21 +00:00
legacy storage type.
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
- **Omnibus installation**
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
```shell
sudo gitlab-rake gitlab:storage:rollback_to_legacy
```
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
- **Source installation**
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
```shell
sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production
```
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
If you have any existing integration, you may want to do a small rollback first,
to validate. You can do so by specifying an ID range with the operation by using
the environment variables `ID_FROM` and `ID_TO` . For example, to limit the rollout
to project IDs 50 to 100 in an Omnibus GitLab installation:
2017-11-18 07:12:37 +00:00
2020-01-30 15:09:15 +00:00
```shell
2020-05-19 12:08:21 +00:00
sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100
2017-11-18 07:12:37 +00:00
```
2020-07-29 00:09:37 +00:00
You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page.
2020-05-19 12:08:21 +00:00
On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish.
2017-11-18 07:12:37 +00:00
2020-08-31 21:10:19 +00:00
After it reaches zero, you can confirm every project has been rolled back by running the commands above.
2020-05-19 12:08:21 +00:00
If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks.
2021-04-26 03:10:04 +00:00
Any error or warning is logged in Sidekiq's log file.
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
If you have a Geo setup, the rollback will not be reflected automatically
on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove
the remaining repositories from the special `@hashed/` folder manually.