f2d50af917
Previously, we stored these as serialised fields - `st_{commits,diffs}` - on the
`merge_request_diffs` table. These now have their own tables -
`merge_request_diff_{commits,diffs}` - with a column for each attribute of the
serialised data.
Add a background migration to go through the existing MR diffs and migrate them
to the new format. Ignore any contents that cannot be displayed. Assuming that
we have 5 million rows to migrate, and each batch of 2,500 rows can be
completed in 5 minutes, this will take about 7 days to migrate everything.
|
||
|---|---|---|
| .. | ||
| active_record | ||
| add_foreign_key_to_merge_requests_spec.rb | ||
| add_head_pipeline_for_each_merge_request_spec.rb | ||
| clean_appearance_symlinks_spec.rb | ||
| clean_stage_id_reference_migration_spec.rb | ||
| clean_upload_symlinks_spec.rb | ||
| cleanup_move_system_upload_folder_symlink_spec.rb | ||
| cleanup_namespaceless_pending_delete_projects_spec.rb | ||
| convert_custom_notification_settings_to_columns_spec.rb | ||
| fix_wrongly_renamed_routes_spec.rb | ||
| migrate_build_stage_reference_again_spec.rb | ||
| migrate_old_artifacts_spec.rb | ||
| migrate_pipeline_stages_spec.rb | ||
| migrate_process_commit_worker_jobs_spec.rb | ||
| migrate_stage_id_reference_in_background_spec.rb | ||
| migrate_user_activities_to_users_last_activity_on_spec.rb | ||
| migrate_user_project_view_spec.rb | ||
| move_personal_snippets_files_spec.rb | ||
| move_system_upload_folder_spec.rb | ||
| move_uploads_to_system_dir_spec.rb | ||
| README.md | ||
| remove_dot_git_from_usernames_spec.rb | ||
| rename_duplicated_variable_key_spec.rb | ||
| rename_more_reserved_project_names_spec.rb | ||
| rename_reserved_project_names_spec.rb | ||
| rename_system_namespaces_spec.rb | ||
| rename_users_with_renamed_namespace_spec.rb | ||
| schedule_merge_request_diff_migrations_spec.rb | ||
| turn_nested_groups_into_regular_groups_for_mysql_spec.rb | ||
| update_retried_for_ci_build_spec.rb | ||
| update_upload_paths_to_system_spec.rb | ||
Testing migrations
In order to reliably test a migration, we need to test it against a database
schema that this migration has been written for. In order to achieve that we
have some migration helpers and RSpec test tag, called :migration.
If you want to write a test for a migration consider adding :migration tag to
the test signature, like describe SomeMigrationClass, :migration.
How does it work?
Adding a :migration tag to a test signature injects a few before / after
hooks to the test.
The most important change is that adding a :migration tag adds a before
hook that will revert all migrations to the point that a migration under test
is not yet migrated.
In other words, our custom RSpec hooks will find a previous migration, and migrate the database down to the previous migration version.
With this approach you can test a migration against a database schema that this migration has been written for.
Use migrate! helper to run the migration that is under test.
The after hook will migrate the database up and reinstitutes the latest
schema version, so that the process does not affect subsequent specs and
ensures proper isolation.
Available helpers
Use table helper to create a temporary ActiveRecord::Base derived model
for a table.
Use migrate! helper to run the migration that is under test. It will not only
run migration, but will also bump the schema version in the schema_migrations
table. It is necessary because in the after hook we trigger the rest of
the migrations, and we need to know where to start.
See spec/support/migrations_helpers.rb for all the available helpers.
An example
require 'spec_helper'
# Load a migration class.
require Rails.root.join('db', 'post_migrate', '20170526185842_migrate_pipeline_stages.rb')
describe MigratePipelineStages, :migration do
# Create test data - pipeline and CI/CD jobs.
let(:jobs) { table(:ci_builds) }
let(:stages) { table(:ci_stages) }
let(:pipelines) { table(:ci_pipelines) }
let(:projects) { table(:projects) }
before do
projects.create!(id: 123, name: 'gitlab1', path: 'gitlab1')
pipelines.create!(id: 1, project_id: 123, ref: 'master', sha: 'adf43c3a')
jobs.create!(id: 1, commit_id: 1, project_id: 123, stage_idx: 2, stage: 'build')
jobs.create!(id: 2, commit_id: 1, project_id: 123, stage_idx: 1, stage: 'test')
end
# Test the migration.
it 'correctly migrates pipeline stages' do
expect(stages.count).to be_zero
migrate!
expect(stages.count).to eq 2
expect(stages.all.pluck(:name)).to match_array %w[test build]
end
end
Best practices
- Use only one test example per migration unless there is a good reason to use more.
- Note that this type of tests do not run within the transaction, we use a truncation database cleanup strategy. Do not depend on transaction being present.