2020-04-28 20:09:38 -04:00
|
|
|
# Uploads migrate Rake tasks **(CORE ONLY)**
|
2018-02-21 11:43:21 -05:00
|
|
|
|
2020-09-04 05:08:38 -04:00
|
|
|
There is a Rake task for migrating uploads between different storage types.
|
|
|
|
|
|
|
|
- Migrate all uploads with [`gitlab:uploads:migrate:all`](#all-in-one-rake-task) or
|
|
|
|
- To only migrate specific upload types, use [`gitlab:uploads:migrate`](#individual-rake-tasks).
|
2018-02-21 11:43:21 -05:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
## Migrate to object storage
|
2018-02-21 11:43:21 -05:00
|
|
|
|
2020-09-07 11:09:04 -04:00
|
|
|
After [configuring the object storage](../../uploads.md#using-object-storage) for GitLab's
|
2020-04-28 02:09:49 -04:00
|
|
|
uploads, use this task to migrate existing uploads from the local storage to the remote storage.
|
|
|
|
|
|
|
|
Read more about using [object storage with GitLab](../../object_storage.md).
|
2018-02-21 11:43:21 -05:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
NOTE: **Note:**
|
|
|
|
All of the processing will be done in a background worker and requires **no downtime**.
|
2020-04-09 11:09:29 -04:00
|
|
|
|
2020-03-31 08:08:09 -04:00
|
|
|
### All-in-one Rake task
|
2018-09-18 10:41:46 -04:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
GitLab provides a wrapper Rake task that migrates all uploaded files (for example avatars, logos,
|
|
|
|
attachments, and favicon) to object storage in one step. The wrapper task invokes individual Rake
|
|
|
|
tasks to migrate files falling under each of these categories one by one.
|
|
|
|
|
|
|
|
These [individual Rake tasks](#individual-rake-tasks) are described in the next section.
|
|
|
|
|
|
|
|
To migrate all uploads from local storage to object storage, run:
|
2018-09-18 10:41:46 -04:00
|
|
|
|
|
|
|
**Omnibus Installation**
|
|
|
|
|
2020-01-30 10:09:15 -05:00
|
|
|
```shell
|
2018-09-18 10:41:46 -04:00
|
|
|
gitlab-rake "gitlab:uploads:migrate:all"
|
|
|
|
```
|
|
|
|
|
|
|
|
**Source Installation**
|
|
|
|
|
2020-01-30 10:09:15 -05:00
|
|
|
```shell
|
2018-09-18 10:41:46 -04:00
|
|
|
sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate:all
|
|
|
|
```
|
|
|
|
|
2020-03-31 08:08:09 -04:00
|
|
|
### Individual Rake tasks
|
2018-02-21 11:43:21 -05:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
If you already ran the [all-in-one Rake task](#all-in-one-rake-task), there is no need to run these
|
|
|
|
individual tasks.
|
|
|
|
|
|
|
|
The Rake task uses three parameters to find uploads to migrate:
|
2018-09-18 10:41:46 -04:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
| Parameter | Type | Description |
|
|
|
|
|:-----------------|:--------------|:-------------------------------------------------------|
|
|
|
|
| `uploader_class` | string | Type of the uploader to migrate from. |
|
|
|
|
| `model_class` | string | Type of the model to migrate from. |
|
|
|
|
| `mount_point` | string/symbol | Name of the model's column the uploader is mounted on. |
|
2018-02-21 11:43:21 -05:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
NOTE: **Note:**
|
|
|
|
These parameters are mainly internal to GitLab's structure, you may want to refer to the task list
|
|
|
|
instead below.
|
2018-02-21 11:43:21 -05:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
This task also accepts an environment variable which you can use to override
|
|
|
|
the default batch size:
|
2018-09-18 10:41:46 -04:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
| Variable | Type | Description |
|
|
|
|
|:---------|:--------|:--------------------------------------------------|
|
|
|
|
| `BATCH` | integer | Specifies the size of the batch. Defaults to 200. |
|
2018-02-21 11:43:21 -05:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
The following shows how to run `gitlab:uploads:migrate` for individual types of uploads.
|
2018-02-21 11:43:21 -05:00
|
|
|
|
2018-09-18 10:41:46 -04:00
|
|
|
**Omnibus Installation**
|
2018-02-21 11:43:21 -05:00
|
|
|
|
2020-01-30 10:09:15 -05:00
|
|
|
```shell
|
2018-02-21 11:43:21 -05:00
|
|
|
# gitlab-rake gitlab:uploads:migrate[uploader_class, model_class, mount_point]
|
|
|
|
|
|
|
|
# Avatars
|
|
|
|
gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]"
|
|
|
|
gitlab-rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]"
|
|
|
|
gitlab-rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]"
|
|
|
|
|
|
|
|
# Attachments
|
|
|
|
gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]"
|
|
|
|
gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]"
|
|
|
|
gitlab-rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]"
|
|
|
|
|
2018-09-18 10:41:46 -04:00
|
|
|
# Favicon
|
|
|
|
gitlab-rake "gitlab:uploads:migrate[FaviconUploader, Appearance, :favicon]"
|
|
|
|
|
2018-02-21 11:43:21 -05:00
|
|
|
# Markdown
|
|
|
|
gitlab-rake "gitlab:uploads:migrate[FileUploader, Project]"
|
|
|
|
gitlab-rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]"
|
|
|
|
gitlab-rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]"
|
|
|
|
gitlab-rake "gitlab:uploads:migrate[FileUploader, MergeRequest]"
|
2020-02-21 16:08:57 -05:00
|
|
|
|
2020-05-15 11:08:04 -04:00
|
|
|
# Design Management design thumbnails
|
2020-02-21 16:08:57 -05:00
|
|
|
gitlab-rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, DesignManagement::Action, :image_v432x230]"
|
2018-02-21 11:43:21 -05:00
|
|
|
```
|
|
|
|
|
|
|
|
**Source Installation**
|
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
NOTE: **Note:**
|
2018-02-21 11:43:21 -05:00
|
|
|
Use `RAILS_ENV=production` for every task.
|
|
|
|
|
2020-01-30 10:09:15 -05:00
|
|
|
```shell
|
2018-02-21 11:43:21 -05:00
|
|
|
# sudo -u git -H bundle exec rake gitlab:uploads:migrate
|
|
|
|
|
|
|
|
# Avatars
|
|
|
|
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Project, :avatar]"
|
|
|
|
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, Group, :avatar]"
|
|
|
|
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AvatarUploader, User, :avatar]"
|
|
|
|
|
|
|
|
# Attachments
|
|
|
|
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Note, :attachment]"
|
|
|
|
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :logo]"
|
|
|
|
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[AttachmentUploader, Appearance, :header_logo]"
|
|
|
|
|
2018-09-18 10:41:46 -04:00
|
|
|
# Favicon
|
|
|
|
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FaviconUploader, Appearance, :favicon]"
|
|
|
|
|
2018-02-21 11:43:21 -05:00
|
|
|
# Markdown
|
|
|
|
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, Project]"
|
|
|
|
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]"
|
|
|
|
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]"
|
|
|
|
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeRequest]"
|
|
|
|
|
2020-05-15 11:08:04 -04:00
|
|
|
# Design Management design thumbnails
|
2020-02-21 16:08:57 -05:00
|
|
|
sudo -u git -H bundle exec rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, DesignManagement::Action]"
|
2018-02-21 11:43:21 -05:00
|
|
|
```
|
2019-08-13 04:30:32 -04:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
## Migrate to local storage
|
2019-09-20 08:05:52 -04:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
If you need to disable [object storage](../../object_storage.md) for any reason, you must first
|
|
|
|
migrate your data out of object storage and back into your local storage.
|
2019-09-20 08:05:52 -04:00
|
|
|
|
|
|
|
CAUTION: **Warning:**
|
2019-11-27 22:06:32 -05:00
|
|
|
**Extended downtime is required** so no new files are created in object storage during
|
|
|
|
the migration. A configuration setting will be added soon to allow migrating
|
|
|
|
from object storage to local files with only a brief moment of downtime for configuration changes.
|
2020-05-21 02:08:25 -04:00
|
|
|
To follow progress, see the [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30979).
|
2019-09-20 08:05:52 -04:00
|
|
|
|
2020-03-31 08:08:09 -04:00
|
|
|
### All-in-one Rake task
|
2019-09-20 08:05:52 -04:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
GitLab provides a wrapper Rake task that migrates all uploaded files (for example, avatars, logos,
|
|
|
|
attachments, and favicon) to local storage in one step. The wrapper task invokes individual Rake
|
|
|
|
tasks to migrate files falling under each of these categories one by one.
|
|
|
|
|
|
|
|
For details on these Rake tasks, refer to [Individual Rake tasks](#individual-rake-tasks),
|
2019-09-20 08:05:52 -04:00
|
|
|
keeping in mind the task name in this case is `gitlab:uploads:migrate_to_local`.
|
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
To migrate uploads from object storage to local storage:
|
2019-09-20 08:05:52 -04:00
|
|
|
|
2020-07-27 11:09:25 -04:00
|
|
|
1. Disable both `direct_upload` and `background_upload` under `uploads` settings in `gitlab.rb`:
|
|
|
|
|
|
|
|
```ruby
|
|
|
|
gitlab_rails['uploads_object_store_direct_upload'] = false
|
|
|
|
gitlab_rails['uploads_object_store_background_upload'] = false
|
|
|
|
```
|
|
|
|
|
|
|
|
Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure).
|
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
1. Run the Rake task:
|
2019-09-20 08:05:52 -04:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
**Omnibus Installation**
|
2019-09-20 08:05:52 -04:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
```shell
|
|
|
|
gitlab-rake "gitlab:uploads:migrate_to_local:all"
|
|
|
|
```
|
|
|
|
|
|
|
|
**Source Installation**
|
|
|
|
|
|
|
|
```shell
|
|
|
|
sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate_to_local:all
|
|
|
|
```
|
2019-09-20 08:05:52 -04:00
|
|
|
|
2020-04-28 02:09:49 -04:00
|
|
|
After running the Rake task, you can disable object storage by undoing the changes described
|
2020-09-07 11:09:04 -04:00
|
|
|
in the instructions to [configure object storage](../../uploads.md#using-object-storage).
|