This adds a Rubocop configuration file specific to
lib/gitlab/background_migrations. This configuration will be used to
(hopefully) make reviewing background migrations easier by enforcing
stricter rules compared to the rest of GitLab. Because this
configuration is directory specific it will only affect background
migrations.
This version does not use transactions, but individual statements. As we have
unique constraints on the target tables for the inserts, we can just ignore
uniqueness violations there (as long as we always insert the same batch size, in
the same order).
This means the spec now must use truncation, not a transaction, as the
uniqueness violation means that the whole transaction for that spec would be
invalid, which isn't what we'd want. In real-world use, this isn't run in a
transaction anyway.
This commit also wraps unhandled exceptions, for easier finding in Sentry, and
logs with a consistent format, for easier searching.
We were hitting the statement timeout for very large MR diffs. Now we insert at
most 1,000 rows to `merge_request_diff_commits` in a single statement, or 100
rows to `merge_request_diff_files`.
Guess the modes based on the following:
1. If the file didn't exist, it's zero.
2. If the diff contains 'Subproject commit', it might be a submodule, so 0600.
3. Otherwise, it's 0644.
This isn't perfect, but it doesn't have to be - it won't change file modes in
the repository.
Previously, we stored these as serialised fields - `st_{commits,diffs}` - on the
`merge_request_diffs` table. These now have their own tables -
`merge_request_diff_{commits,diffs}` - with a column for each attribute of the
serialised data.
Add a background migration to go through the existing MR diffs and migrate them
to the new format. Ignore any contents that cannot be displayed. Assuming that
we have 5 million rows to migrate, and each batch of 2,500 rows can be
completed in 5 minutes, this will take about 7 days to migrate everything.
Yorick Peterse