2020-10-24 00:08:35 +00:00
---
2020-12-01 18:09:42 +00:00
stage: Enablement
group: Distribution
2020-11-26 06:09:20 +00:00
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-10-24 00:08:35 +00:00
---
2019-07-08 08:50:38 +00:00
# Pseudonymizer **(ULTIMATE)**
2019-05-05 15:21:25 +00:00
2020-04-21 15:21:10 +00:00
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5532) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.1.
2019-05-05 15:21:25 +00:00
2020-12-16 00:09:58 +00:00
As the GitLab database hosts sensitive information, using it unfiltered for analytics
2019-05-05 15:21:25 +00:00
implies high security requirements. To help alleviate this constraint, the Pseudonymizer
2020-12-16 00:09:58 +00:00
service is used to export GitLab data in a pseudonymized way.
2019-05-05 15:21:25 +00:00
2020-12-04 21:09:29 +00:00
WARNING:
2019-05-05 15:21:25 +00:00
This process is not impervious. If the source data is available, it's possible for
a user to correlate data to the pseudonymized version.
The Pseudonymizer currently uses `HMAC(SHA256)` to mutate fields that shouldn't
be textually exported. This ensures that:
- the end-user of the data source cannot infer/revert the pseudonymized fields
- the referential integrity is maintained
## Configuration
2021-03-03 12:11:16 +00:00
To configure the Pseudonymizer, you need to:
2019-05-05 15:21:25 +00:00
- Provide a manifest file that describes which fields should be included or
2019-09-18 14:02:45 +00:00
pseudonymized ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab/tree/master/config/pseudonymizer.yml)).
2020-11-19 21:09:07 +00:00
A default manifest is provided with the GitLab installation, using a relative file path that resolves from the Rails root.
2019-05-05 15:21:25 +00:00
Alternatively, you can use an absolute file path.
2019-07-09 03:28:41 +00:00
- Use an object storage and specify the connection parameters in the `pseudonymizer.upload.connection` configuration option.
2019-05-05 15:21:25 +00:00
2020-04-09 15:09:29 +00:00
[Read more about using object storage with GitLab ](object_storage.md ).
2019-05-05 15:21:25 +00:00
**For Omnibus installations:**
1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with
the values you want:
2019-07-09 07:16:17 +00:00
```ruby
gitlab_rails['pseudonymizer_manifest'] = 'config/pseudonymizer.yml'
gitlab_rails['pseudonymizer_upload_remote_directory'] = 'gitlab-elt' # bucket name
gitlab_rails['pseudonymizer_upload_connection'] = {
'provider' => 'AWS',
'region' => 'eu-central-1',
'aws_access_key_id' => 'AWS_ACCESS_KEY_ID',
'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY'
}
```
If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs.
```ruby
gitlab_rails['pseudonymizer_upload_connection'] = {
'provider' => 'AWS',
'region' => 'eu-central-1',
'use_iam_profile' => true
}
```
2019-05-05 15:21:25 +00:00
1. Save the file and [reconfigure GitLab ](restart_gitlab.md#omnibus-gitlab-reconfigure )
for the changes to take effect.
---
**For installations from source:**
1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following
lines:
2019-07-09 07:16:17 +00:00
```yaml
pseudonymizer:
2019-07-11 22:53:54 +00:00
manifest: config/pseudonymizer.yml
upload:
2019-07-09 07:16:17 +00:00
remote_directory: 'gitlab-elt' # bucket name
connection:
provider: AWS
aws_access_key_id: AWS_ACCESS_KEY_ID
aws_secret_access_key: AWS_SECRET_ACCESS_KEY
region: eu-central-1
```
2019-05-05 15:21:25 +00:00
1. Save the file and [restart GitLab ](restart_gitlab.md#installations-from-source )
for the changes to take effect.
## Usage
2021-03-03 12:11:16 +00:00
You can optionally run the Pseudonymizer using the following environment variables:
2019-05-05 15:21:25 +00:00
- `PSEUDONYMIZER_OUTPUT_DIR` - where to store the output CSV files (defaults to `/tmp` )
- `PSEUDONYMIZER_BATCH` - the batch size when querying the DB (defaults to `100000` )
2020-01-30 15:09:15 +00:00
```shell
2019-05-05 15:21:25 +00:00
## Omnibus
sudo gitlab-rake gitlab:db:pseudonymizer
## Source
sudo -u git -H bundle exec rake gitlab:db:pseudonymizer RAILS_ENV=production
```
2020-11-19 21:09:07 +00:00
This produces some CSV files that might be very large, so make sure the
2019-05-05 15:21:25 +00:00
`PSEUDONYMIZER_OUTPUT_DIR` has sufficient space. As a rule of thumb, at least
10% of the database size is recommended.
After the pseudonymizer has run, the output CSV files should be uploaded to the
configured object storage and deleted from the local disk.