2020-10-24 00:08:35 +00:00
---
2022-05-30 00:08:35 +00:00
stage: Systems
2020-12-01 18:09:42 +00:00
group: Distribution
2022-09-21 21:13:33 +00:00
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
2020-10-24 00:08:35 +00:00
---
2021-01-28 06:08:59 +00:00
# Repository storage Rake tasks **(FREE SELF)**
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
This is a collection of Rake tasks to help you list and migrate
existing projects and their attachments to the new
[hashed storage ](../repository_storage_types.md ) that GitLab
uses to organize the Git data.
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
## List projects and attachments
2017-09-06 05:16:26 +00:00
2021-12-03 06:10:53 +00:00
The following Rake tasks lists the projects and attachments that are
2020-05-19 12:08:21 +00:00
available on legacy and hashed storage.
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
### On legacy storage
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
To have a summary and then a list of projects and their attachments using legacy storage:
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
- **Omnibus installation**
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
```shell
# Projects
sudo gitlab-rake gitlab:storage:legacy_projects
sudo gitlab-rake gitlab:storage:list_legacy_projects
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
# Attachments
sudo gitlab-rake gitlab:storage:legacy_attachments
sudo gitlab-rake gitlab:storage:list_legacy_attachments
```
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
- **Source installation**
2018-06-07 15:40:44 +00:00
2020-05-19 12:08:21 +00:00
```shell
# Projects
sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_projects RAILS_ENV=production
2018-06-07 15:40:44 +00:00
2020-05-19 12:08:21 +00:00
# Attachments
sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_legacy_attachments RAILS_ENV=production
```
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
### On hashed storage
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
To have a summary and then a list of projects and their attachments using hashed storage:
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
- **Omnibus installation**
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
```shell
# Projects
sudo gitlab-rake gitlab:storage:hashed_projects
sudo gitlab-rake gitlab:storage:list_hashed_projects
2019-05-05 15:21:25 +00:00
2020-05-19 12:08:21 +00:00
# Attachments
sudo gitlab-rake gitlab:storage:hashed_attachments
sudo gitlab-rake gitlab:storage:list_hashed_attachments
```
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
- **Source installation**
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
```shell
# Projects
sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_projects RAILS_ENV=production
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
# Attachments
sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production
sudo -u git -H bundle exec rake gitlab:storage:list_hashed_attachments RAILS_ENV=production
```
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
## Migrate to hashed storage
2019-03-11 16:38:19 +00:00
2020-12-08 00:09:45 +00:00
WARNING:
2020-05-19 12:08:21 +00:00
In GitLab 13.0, [hashed storage ](../repository_storage_types.md#hashed-storage )
is enabled by default and the legacy storage is deprecated.
2021-04-26 03:10:04 +00:00
GitLab 14.0 eliminates support for legacy storage. If you're on GitLab
2020-05-19 12:08:21 +00:00
13.0 and later, switching new projects to legacy storage is not possible.
2022-05-11 15:07:26 +00:00
The option to choose between hashed and legacy storage in the Admin Area has
2020-05-19 12:08:21 +00:00
been disabled.
2019-03-11 16:38:19 +00:00
2021-12-03 06:10:53 +00:00
This task must be run on any machine that has Rails/Sidekiq configured, and the task
schedules all your existing projects and attachments associated with it to be
2020-12-10 12:09:43 +00:00
migrated to the **Hashed** storage type:
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
- **Omnibus installation**
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
```shell
sudo gitlab-rake gitlab:storage:migrate_to_hashed
```
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
- **Source installation**
2019-03-11 16:38:19 +00:00
2020-05-19 12:08:21 +00:00
```shell
sudo -u git -H bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production
```
If you have any existing integration, you may want to do a small rollout first,
to validate. You can do so by specifying an ID range with the operation by using
the environment variables `ID_FROM` and `ID_TO` . For example, to limit the rollout
to project IDs 50 to 100 in an Omnibus GitLab installation:
2019-03-11 16:38:19 +00:00
2020-01-30 15:09:15 +00:00
```shell
2020-05-19 12:08:21 +00:00
sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100
2019-03-11 16:38:19 +00:00
```
2021-06-18 15:10:16 +00:00
To monitor the progress in GitLab:
2019-03-11 16:38:19 +00:00
2022-09-14 18:12:48 +00:00
1. On the top bar, select **Main menu > Admin** .
2021-06-18 15:10:16 +00:00
1. On the left sidebar, select **Monitoring > Background Jobs** .
1. Watch how long the `hashed_storage:hashed_storage_project_migrate` queue
2021-12-03 06:10:53 +00:00
takes to finish. After it reaches zero, you can confirm every project
2021-06-18 15:10:16 +00:00
has been migrated by running the commands above.
If you find it necessary, you can run the previous migration script again to schedule missing projects.
2019-03-11 16:38:19 +00:00
2021-04-26 03:10:04 +00:00
Any error or warning is logged in Sidekiq's log file.
2019-03-11 16:38:19 +00:00
2020-09-15 15:10:08 +00:00
If [Geo ](../geo/index.md ) is enabled, each project that is successfully migrated
2020-05-19 12:08:21 +00:00
generates an event to replicate the changes on any **secondary** nodes.
2017-11-18 07:12:37 +00:00
2021-04-26 21:10:25 +00:00
You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but there are
2021-06-18 15:10:16 +00:00
[additional commands ](#list-projects-and-attachments ) to help you inspect projects and attachments in both legacy and hashed storage.
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
## Rollback from hashed storage to legacy storage
2017-09-06 05:16:26 +00:00
2020-12-08 00:09:45 +00:00
WARNING:
2020-05-19 12:08:21 +00:00
In GitLab 13.0, [hashed storage ](../repository_storage_types.md#hashed-storage )
is enabled by default and the legacy storage is deprecated.
2021-04-26 03:10:04 +00:00
GitLab 14.0 eliminates support for legacy storage. If you're on GitLab
2020-05-19 12:08:21 +00:00
13.0 and later, switching new projects to legacy storage is not possible.
2022-05-11 15:07:26 +00:00
The option to choose between hashed and legacy storage in the Admin Area has
2020-05-19 12:08:21 +00:00
been disabled.
2017-09-06 05:16:26 +00:00
2021-04-26 03:10:04 +00:00
This task schedules all your existing projects and associated attachments to be rolled back to the
2020-05-19 12:08:21 +00:00
legacy storage type.
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
- **Omnibus installation**
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
```shell
sudo gitlab-rake gitlab:storage:rollback_to_legacy
```
2017-09-06 05:16:26 +00:00
2020-05-19 12:08:21 +00:00
- **Source installation**
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
```shell
sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production
```
2017-11-18 07:12:37 +00:00
2020-05-19 12:08:21 +00:00
If you have any existing integration, you may want to do a small rollback first,
to validate. You can do so by specifying an ID range with the operation by using
the environment variables `ID_FROM` and `ID_TO` . For example, to limit the rollout
to project IDs 50 to 100 in an Omnibus GitLab installation:
2017-11-18 07:12:37 +00:00
2020-01-30 15:09:15 +00:00
```shell
2020-05-19 12:08:21 +00:00
sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100
2017-11-18 07:12:37 +00:00
```
2020-07-29 00:09:37 +00:00
You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page.
2021-12-03 06:10:53 +00:00
On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process takes to finish.
2017-11-18 07:12:37 +00:00
2020-08-31 21:10:19 +00:00
After it reaches zero, you can confirm every project has been rolled back by running the commands above.
2020-05-19 12:08:21 +00:00
If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks.
2021-04-26 03:10:04 +00:00
Any error or warning is logged in Sidekiq's log file.
2017-11-18 07:12:37 +00:00
2021-12-03 06:10:53 +00:00
If you have a Geo setup, the rollback is not reflected automatically
2020-05-19 12:08:21 +00:00
on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove
the remaining repositories from the special `@hashed/` folder manually.
2022-05-04 15:09:12 +00:00
## Troubleshooting
The Rake task might not be able to complete the migration to hashed storage.
Checks on the instance will continue to report that there is legacy data:
```plaintext
* Found 1 projects using Legacy Storage
- janedoe/testproject (id: 1234)
```
If you have a subscription, [raise a ticket with GitLab support ](https://support.gitlab.com )
as most of the fixes are relatively high risk, involving running code on the Rails console.
### Read only projects
2022-10-17 12:10:08 +00:00
If you have set projects as read only they might fail to migrate.
2022-05-04 15:09:12 +00:00
1. [Start a Rails console ](../operations/rails_console.md#starting-a-rails-console-session ).
1. Check if the project is read only:
```ruby
project = Project.find_by_full_path('janedoe/testproject')
project.repository_read_only
```
1. If it returns `true` (not `nil` or `false` ), set it writable:
```ruby
project.update!(repository_read_only: false)
```
1. [Re-run the migration Rake task ](#migrate-to-hashed-storage ).
1. Set the project read-only again:
```ruby
project.update!(repository_read_only: true)
```
### Projects pending deletion
2022-05-11 15:07:26 +00:00
Check the project details in the Admin Area. If deleting the project failed
2022-05-04 15:09:12 +00:00
it will show as `Marked For Deletion At ..` , `Scheduled Deletion At ..` and
`pending removal` , but the dates will not be recent.
Delete the project using the Rails console:
1. [Start a Rails console ](../operations/rails_console.md#starting-a-rails-console-session ).
1. With the following code, select the project to be deleted and account to action it:
```ruby
project = Project.find_by_full_path('janedoe/testproject')
user = User.find_by_username('admin_handle')
puts "\nproject selected for deletion is:\nID: #{project.id}\nPATH: #{project.full_path}\nNAME: #{project.name}\n\n"
```
2022-05-11 15:07:26 +00:00
- Replace `janedoe/testproject` with your project path from the Rake take output or from the Admin Area.
2022-05-04 15:09:12 +00:00
- Replace `admin_handle` with the handle of an instance administrator or with `root` .
- Verify the output before proceeding. **There are no other checks performed** .
2022-10-17 12:10:08 +00:00
1. [Destroy the project ](../../user/project/working_with_projects.md#delete-a-project-using-console ) **immediately** :
2022-05-04 15:09:12 +00:00
```ruby
Projects::DestroyService.new(project, user).execute
```
If destroying the project generates a stack trace relating to encryption or the error `OpenSSL::Cipher::CipherError` :
1. [Verify your GitLab secrets ](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets ).
1. If the affected projects have secrets that cannot be decrypted it will be necessary to remove those specific secrets.
[Our documentation for dealing with lost secrets ](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost )
is for loss of all secrets, but it's possible for specific projects to be affected. For example,
to [reset specific runner registration tokens ](../../raketasks/backup_restore.md#reset-runner-registration-tokens )
for a specific project ID:
```sql
UPDATE projects SET runners_token = null, runners_token_encrypted = null where id = 1234;
```
### `Repository cannot be moved from` errors in Sidekiq log
Projects might fail to migrate with errors in the Sidekiq log:
```shell
# grep 'Repository cannot be moved' /var/log/gitlab/sidekiq/current
{"severity":"ERROR","time":"2021-02-29T02:29:02.021Z","message":"Repository cannot be moved from 'janedoe/testproject' to '@hashed< value > ' (PROJECT_ID=1234)"}
```
This might be caused by [a bug ](https://gitlab.com/gitlab-org/gitlab/-/issues/259605 ) in the original code for hashed storage migration.
[There is a workaround for projects still affected by this issue ](https://gitlab.com/-/snippets/2039252 ).