gitlab-org--gitlab-foss/doc/raketasks/cleanup.md

# Cleanup

## Remove garbage from filesystem. Important! Data loss!

Remove namespaces(dirs) from all repository storage paths if they don't exist in GitLab database.

```
# omnibus-gitlab
sudo gitlab-rake gitlab:cleanup:dirs

# installation from source
bundle exec rake gitlab:cleanup:dirs RAILS_ENV=production
```

Rename repositories from all repository storage paths if they don't exist in GitLab database.
The repositories get a `+orphaned+TIMESTAMP` suffix so that they cannot block new repositories from being created.

```
# omnibus-gitlab
sudo gitlab-rake gitlab:cleanup:repos

# installation from source
bundle exec rake gitlab:cleanup:repos RAILS_ENV=production
```

Remove old repository copies from repositories moved to another storage.

```
# omnibus-gitlab
sudo gitlab-rake gitlab:cleanup:moved

# installation from source
bundle exec rake gitlab:cleanup:moved RAILS_ENV=production
```

Clean up local project upload files if they don't exist in GitLab database. The
task attempts to fix the file if it can find its project, otherwise it moves the
file to a lost and found directory.

```
# omnibus-gitlab
sudo gitlab-rake gitlab:cleanup:project_uploads

# installation from source
bundle exec rake gitlab:cleanup:project_uploads RAILS_ENV=production
```

Example output:

```
$ sudo gitlab-rake gitlab:cleanup:project_uploads
I, [2018-07-27T12:08:27.671559 #89817]  INFO -- : Looking for orphaned project uploads to clean up. Dry run...
D, [2018-07-27T12:08:28.293568 #89817] DEBUG -- : Processing batch of 500 project upload file paths, starting with /opt/gitlab/embedded/service/gitlab-rails/public/uploads/test.out
I, [2018-07-27T12:08:28.689869 #89817]  INFO -- : Can move to lost and found /opt/gitlab/embedded/service/gitlab-rails/public/uploads/test.out -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/project-lost-found/test.out
I, [2018-07-27T12:08:28.755624 #89817]  INFO -- : Can fix /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/89a0f7b0b97008a4a18cedccfdcd93fb/foo.txt -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/qux/foo/bar/89a0f7b0b97008a4a18cedccfdcd93fb/foo.txt
I, [2018-07-27T12:08:28.760257 #89817]  INFO -- : Can move to lost and found /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/1dd6f0f7eefd2acc4c2233f89a0f7b0b/image.png -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/project-lost-found/foo/bar/1dd6f0f7eefd2acc4c2233f89a0f7b0b/image.png
I, [2018-07-27T12:08:28.764470 #89817]  INFO -- : To cleanup these files run this command with DRY_RUN=false

$ sudo gitlab-rake gitlab:cleanup:project_uploads DRY_RUN=false
I, [2018-07-27T12:08:32.944414 #89936]  INFO -- : Looking for orphaned project uploads to clean up...
D, [2018-07-27T12:08:33.293568 #89817] DEBUG -- : Processing batch of 500 project upload file paths, starting with /opt/gitlab/embedded/service/gitlab-rails/public/uploads/test.out
I, [2018-07-27T12:08:33.689869 #89817]  INFO -- : Did move to lost and found /opt/gitlab/embedded/service/gitlab-rails/public/uploads/test.out -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/project-lost-found/test.out
I, [2018-07-27T12:08:33.755624 #89817]  INFO -- : Did fix /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/89a0f7b0b97008a4a18cedccfdcd93fb/foo.txt -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/qux/foo/bar/89a0f7b0b97008a4a18cedccfdcd93fb/foo.txt
I, [2018-07-27T12:08:33.760257 #89817]  INFO -- : Did move to lost and found /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/1dd6f0f7eefd2acc4c2233f89a0f7b0b/image.png -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/project-lost-found/foo/bar/1dd6f0f7eefd2acc4c2233f89a0f7b0b/image.png
```

Remove object store upload files if they don't exist in GitLab database.

```
# omnibus-gitlab
sudo gitlab-rake gitlab:cleanup:remote_upload_files

# installation from source
bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production
```

Example output:

```
$ sudo gitlab-rake gitlab:cleanup:remote_upload_files

I, [2018-08-02T10:26:13.995978 #45011]  INFO -- : Looking for orphaned remote uploads to remove. Dry run...
I, [2018-08-02T10:26:14.120400 #45011]  INFO -- : Can be moved to lost and found: @hashed/6b/DSC_6152.JPG
I, [2018-08-02T10:26:14.120482 #45011]  INFO -- : Can be moved to lost and found: @hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451/711491b29d3eb08837798c4909e2aa4d/DSC00314.jpg
I, [2018-08-02T10:26:14.120634 #45011]  INFO -- : To cleanup these files run this command with DRY_RUN=false
```

```
$ sudo gitlab-rake gitlab:cleanup:remote_upload_files DRY_RUN=false

I, [2018-08-02T10:26:47.598424 #45087]  INFO -- : Looking for orphaned remote uploads to remove...
I, [2018-08-02T10:26:47.753131 #45087]  INFO -- : Moved to lost and found: @hashed/6b/DSC_6152.JPG -> lost_and_found/@hashed/6b/DSC_6152.JPG
I, [2018-08-02T10:26:47.764356 #45087]  INFO -- : Moved to lost and found: @hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451/711491b29d3eb08837798c4909e2aa4d/DSC00314.jpg -> lost_and_found/@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451/711491b29d3eb08837798c4909e2aa4d/DSC00314.jpg
```

## Remove orphan artifact files

When you notice there are more job artifacts files on disk than there
should be, you can run:

```shell
gitlab-rake gitlab:cleanup:orphan_job_artifact_files
```

This command:

- Scans through the entire artifacts folder.
- Checks which files still have a record in the database.
- If no database record is found, the file is deleted from disk.

By default, this task does not delete anything but shows what it can
delete. Run the command with `DRY_RUN=false` if you actually want to
delete the files:

```shell
gitlab-rake gitlab:cleanup:orphan_job_artifact_files DRY_RUN=false
```

You can also limit the number of files to delete with `LIMIT`:

```shell
gitlab-rake gitlab:cleanup:orphan_job_artifact_files LIMIT=100`
```

This will only delete up to 100 files from disk. You can use this to
delete a small set for testing purposes.

If you provide `DEBUG=1`, you'll see the full path of every file that
is detected as being an orphan.

If `ionice` is installed, the tasks uses it to ensure the command is
not causing too much load on the disk. You can configure the niceness
level with `NICENESS`. Below are the valid levels, but consult
`man 1 ionice` to be sure.

- `0` or `None`
- `1` or `Realtime`
- `2` or `Best-effort` (default)
- `3` or `Idle`

## Remove expired ActiveSession lookup keys

```
# omnibus-gitlab
sudo gitlab-rake gitlab:cleanup:sessions:active_sessions_lookup_keys

# installation from source
bundle exec rake gitlab:cleanup:sessions:active_sessions_lookup_keys RAILS_ENV=production
```
Add titles to doc pages. 2014-05-27 08:12:15 -04:00			`# Cleanup`

Update docs to markdown style guide. 2014-04-24 18:48:22 -04:00			`## Remove garbage from filesystem. Important! Data loss!`
Added cleanup to help rake 2013-01-05 17:30:38 -05:00
Update documentation according to the new multiple git mount points feature 2016-07-21 17:48:57 -04:00			`Remove namespaces(dirs) from all repository storage paths if they don't exist in GitLab database.`
Added cleanup to help rake 2013-01-05 17:30:38 -05:00
			```
Spell out rake tasks for omnibus-gitlab 2014-06-23 04:38:22 -04:00			`# omnibus-gitlab`
			`sudo gitlab-rake gitlab:cleanup:dirs`

Drop comments on cookbook installing from source now that it uses Omnibus. 2015-01-28 01:50:31 -05:00			`# installation from source`
Added cleanup to help rake 2013-01-05 17:30:38 -05:00			`bundle exec rake gitlab:cleanup:dirs RAILS_ENV=production`
			```

Update documentation according to the new multiple git mount points feature 2016-07-21 17:48:57 -04:00			`Rename repositories from all repository storage paths if they don't exist in GitLab database.`
Improve 'rake gitlab:cleanup:repos' documentation 2015-09-25 08:38:42 -04:00			The repositories get a `+orphaned+TIMESTAMP` suffix so that they cannot block new repositories from being created.
Added cleanup to help rake 2013-01-05 17:30:38 -05:00
			```
Spell out rake tasks for omnibus-gitlab 2014-06-23 04:38:22 -04:00			`# omnibus-gitlab`
			`sudo gitlab-rake gitlab:cleanup:repos`

Drop comments on cookbook installing from source now that it uses Omnibus. 2015-01-28 01:50:31 -05:00			`# installation from source`
Added cleanup to help rake 2013-01-05 17:30:38 -05:00			`bundle exec rake gitlab:cleanup:repos RAILS_ENV=production`
			```
Add local project uploads cleanup task 2018-07-26 17:23:33 -04:00
Docs: Merge Various EE /doc dirs to CE 2019-05-05 11:59:22 -04:00			`Remove old repository copies from repositories moved to another storage.`

			```
			`# omnibus-gitlab`
			`sudo gitlab-rake gitlab:cleanup:moved`

			`# installation from source`
			`bundle exec rake gitlab:cleanup:moved RAILS_ENV=production`
			```

Add local project uploads cleanup task 2018-07-26 17:23:33 -04:00			`Clean up local project upload files if they don't exist in GitLab database. The`
			`task attempts to fix the file if it can find its project, otherwise it moves the`
			`file to a lost and found directory.`

			```
			`# omnibus-gitlab`
			`sudo gitlab-rake gitlab:cleanup:project_uploads`

			`# installation from source`
			`bundle exec rake gitlab:cleanup:project_uploads RAILS_ENV=production`
			```

			`Example output:`

			```
			`$ sudo gitlab-rake gitlab:cleanup:project_uploads`
			`I, [2018-07-27T12:08:27.671559 #89817] INFO -- : Looking for orphaned project uploads to clean up. Dry run...`
			`D, [2018-07-27T12:08:28.293568 #89817] DEBUG -- : Processing batch of 500 project upload file paths, starting with /opt/gitlab/embedded/service/gitlab-rails/public/uploads/test.out`
			`I, [2018-07-27T12:08:28.689869 #89817] INFO -- : Can move to lost and found /opt/gitlab/embedded/service/gitlab-rails/public/uploads/test.out -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/project-lost-found/test.out`
			`I, [2018-07-27T12:08:28.755624 #89817] INFO -- : Can fix /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/89a0f7b0b97008a4a18cedccfdcd93fb/foo.txt -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/qux/foo/bar/89a0f7b0b97008a4a18cedccfdcd93fb/foo.txt`
			`I, [2018-07-27T12:08:28.760257 #89817] INFO -- : Can move to lost and found /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/1dd6f0f7eefd2acc4c2233f89a0f7b0b/image.png -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/project-lost-found/foo/bar/1dd6f0f7eefd2acc4c2233f89a0f7b0b/image.png`
			`I, [2018-07-27T12:08:28.764470 #89817] INFO -- : To cleanup these files run this command with DRY_RUN=false`

			`$ sudo gitlab-rake gitlab:cleanup:project_uploads DRY_RUN=false`
			`I, [2018-07-27T12:08:32.944414 #89936] INFO -- : Looking for orphaned project uploads to clean up...`
			`D, [2018-07-27T12:08:33.293568 #89817] DEBUG -- : Processing batch of 500 project upload file paths, starting with /opt/gitlab/embedded/service/gitlab-rails/public/uploads/test.out`
			`I, [2018-07-27T12:08:33.689869 #89817] INFO -- : Did move to lost and found /opt/gitlab/embedded/service/gitlab-rails/public/uploads/test.out -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/project-lost-found/test.out`
			`I, [2018-07-27T12:08:33.755624 #89817] INFO -- : Did fix /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/89a0f7b0b97008a4a18cedccfdcd93fb/foo.txt -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/qux/foo/bar/89a0f7b0b97008a4a18cedccfdcd93fb/foo.txt`
			`I, [2018-07-27T12:08:33.760257 #89817] INFO -- : Did move to lost and found /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/1dd6f0f7eefd2acc4c2233f89a0f7b0b/image.png -> /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/project-lost-found/foo/bar/1dd6f0f7eefd2acc4c2233f89a0f7b0b/image.png`
Clean orphaned files in object storage 2018-07-30 14:14:38 -04:00			```

			`Remove object store upload files if they don't exist in GitLab database.`

			```
			`# omnibus-gitlab`
			`sudo gitlab-rake gitlab:cleanup:remote_upload_files`

			`# installation from source`
			`bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production`
			```

			`Example output:`

			```
			`$ sudo gitlab-rake gitlab:cleanup:remote_upload_files`

			`I, [2018-08-02T10:26:13.995978 #45011] INFO -- : Looking for orphaned remote uploads to remove. Dry run...`
			`I, [2018-08-02T10:26:14.120400 #45011] INFO -- : Can be moved to lost and found: @hashed/6b/DSC_6152.JPG`
			`I, [2018-08-02T10:26:14.120482 #45011] INFO -- : Can be moved to lost and found: @hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451/711491b29d3eb08837798c4909e2aa4d/DSC00314.jpg`
			`I, [2018-08-02T10:26:14.120634 #45011] INFO -- : To cleanup these files run this command with DRY_RUN=false`
			```

			```
			`$ sudo gitlab-rake gitlab:cleanup:remote_upload_files DRY_RUN=false`

			`I, [2018-08-02T10:26:47.598424 #45087] INFO -- : Looking for orphaned remote uploads to remove...`
			`I, [2018-08-02T10:26:47.753131 #45087] INFO -- : Moved to lost and found: @hashed/6b/DSC_6152.JPG -> lost_and_found/@hashed/6b/DSC_6152.JPG`
			`I, [2018-08-02T10:26:47.764356 #45087] INFO -- : Moved to lost and found: @hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451/711491b29d3eb08837798c4909e2aa4d/DSC00314.jpg -> lost_and_found/@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451/711491b29d3eb08837798c4909e2aa4d/DSC00314.jpg`
			```
Add rake task to clean orphan artifact files This adds the rake task rake gitlab:cleanup:orphan_job_artifact_files. This rake task cleans all orphan job artifact files it can find on disk. It performs a search on the complete folder of all artifacts on disk. Then it filters out all the job artifact ID for which it could not find a record with matching ID in the database. For these, the file is deleted from disk. 2019-06-13 17:07:59 -04:00
			`## Remove orphan artifact files`

			`When you notice there are more job artifacts files on disk than there`
			`should be, you can run:`

			```shell
			`gitlab-rake gitlab:cleanup:orphan_job_artifact_files`
			```

			`This command:`

			`- Scans through the entire artifacts folder.`
			`- Checks which files still have a record in the database.`
			`- If no database record is found, the file is deleted from disk.`

			`By default, this task does not delete anything but shows what it can`
			delete. Run the command with `DRY_RUN=false` if you actually want to
			`delete the files:`

			```shell
			`gitlab-rake gitlab:cleanup:orphan_job_artifact_files DRY_RUN=false`
			```

			You can also limit the number of files to delete with `LIMIT`:

			```shell
			gitlab-rake gitlab:cleanup:orphan_job_artifact_files LIMIT=100`
			```

			`This will only delete up to 100 files from disk. You can use this to`
			`delete a small set for testing purposes.`

			If you provide `DEBUG=1`, you'll see the full path of every file that
			`is detected as being an orphan.`

			If `ionice` is installed, the tasks uses it to ensure the command is
			`not causing too much load on the disk. You can configure the niceness`
			level with `NICENESS`. Below are the valid levels, but consult
			`man 1 ionice` to be sure.

			- `0` or `None`
			- `1` or `Realtime`
			- `2` or `Best-effort` (default)
			- `3` or `Idle`
Rake task to cleanup expired ActiveSession lookup keys In some cases ActiveSession.cleanup was not called after authentication, so for some user ActiveSession lookup keys grew without ever cleaning up. This Rake task manually iterates over the lookup keys and removes ones without existing ActiveSession. 2019-07-12 08:25:12 -04:00
			`## Remove expired ActiveSession lookup keys`

			```
			`# omnibus-gitlab`
			`sudo gitlab-rake gitlab:cleanup:sessions:active_sessions_lookup_keys`

			`# installation from source`
			`bundle exec rake gitlab:cleanup:sessions:active_sessions_lookup_keys RAILS_ENV=production`
			```