gitlab-org--gitlab-foss/spec/migrations
Shinya Maeda ee79ee9473 Cleanup legacy artifact background migration
Add migration and spec

commit 3cc12e1268a6865f524d8fab1804f018312fdf5a
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Tue Jan 8 19:34:31 2019 +0900

    Add changelog to this change

commit 5006fc96e38db514956a35f53ae8ee536548a2e9
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Tue Jan 8 19:32:56 2019 +0900

    Remove partial index from ci_builds artifact_file

    ok

    Update schema

commit 3c956bdc02b195bc67d0327bf3748a631ea5466d
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Mon Jan 7 21:41:27 2019 +0900

    Add frozen_string_literal: true

commit 8c827cd616890160e6e8908843403a6f20c03236
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Mon Jan 7 21:40:06 2019 +0900

    Set batch size 100

commit aeee559777d3bdeadfd2b9bb61d460f2dc1fa8a6
Author: Shinya Maeda <shinya@gitlab.com>
Date:   Fri Jan 4 18:33:05 2019 +0900

    Cleanup legacy artifact background migration
2019-01-17 14:01:50 +09:00
..
active_record Replace 'Sidekiq::Testing.inline!' with 'perform_enqueued_jobs' 2018-07-23 15:34:54 +11:00
add_foreign_key_from_notification_settings_to_users_spec.rb Resolve "Remove ghost notification settings for groups and projects" 2018-08-02 09:34:44 +00:00
add_foreign_key_to_merge_requests_spec.rb
add_foreign_keys_to_todos_spec.rb Fix RSpec/FactoriesInMigrationSpecs in two files 2019-01-15 19:08:06 +01:00
add_head_pipeline_for_each_merge_request_spec.rb
add_not_null_constraint_to_project_mirror_data_foreign_key_spec.rb Adds not null constraint to ProjectMirrorData's foreign_key project_id 2018-05-08 11:25:12 +01:00
add_pages_access_level_to_project_feature_spec.rb Make GitLab pages support access control 2018-10-05 13:41:11 +00:00
add_pipeline_build_foreign_key_spec.rb Remove orphaned builds without project / pipeline reference 2018-05-08 17:12:29 +02:00
add_unique_constraint_to_project_features_project_id_spec.rb Add a unique and not null constraint on the project_features.project_id column 2018-05-15 14:20:14 +02:00
assure_commits_count_for_merge_request_diff_spec.rb
backfill_releases_name_with_tag_name_spec.rb Add name, author and sha to releases 2018-12-13 12:15:21 +01:00
backfill_store_project_full_path_in_repo_spec.rb Always run CleanUp before writing the git config 2018-11-27 22:48:55 +01:00
calculate_conv_dev_index_percentages_spec.rb
change_default_value_for_dsa_key_restriction_spec.rb Add migration to disable the usage of DSA keys 2018-06-04 23:00:25 -05:00
clean_appearance_symlinks_spec.rb
clean_stage_id_reference_migration_spec.rb
clean_stages_statuses_migration_spec.rb
clean_up_for_members_spec.rb Backport changes from spec/migrations/clean_up_for_members_spec.rb 2018-11-27 17:40:14 -05:00
clean_upload_symlinks_spec.rb
cleanup_build_stage_migration_spec.rb Cleanup pipeline build stage background migration 2018-05-07 14:38:11 +02:00
cleanup_environments_external_url_spec.rb Merge branch 'security-stored-xss-for-environments' into 'master' 2018-11-28 19:07:29 -05:00
cleanup_legacy_artifact_migration_spec.rb Cleanup legacy artifact background migration 2019-01-17 14:01:50 +09:00
cleanup_move_system_upload_folder_symlink_spec.rb
cleanup_namespaceless_pending_delete_projects_spec.rb Fix RSpec/FactoriesInMigrationSpecs in two files 2019-01-15 19:08:06 +01:00
cleanup_nonexisting_namespace_pending_delete_projects_spec.rb
cleanup_stages_position_migration_spec.rb Add pipeline stages position clean-up migration 2018-06-29 13:57:52 +00:00
convert_custom_notification_settings_to_columns_spec.rb
create_missing_namespace_for_internal_users_spec.rb
delete_conflicting_redirect_routes_spec.rb
delete_inconsistent_internal_id_records_spec.rb Add config_options|variables to BuildMetadata 2019-01-04 16:38:17 +01:00
drop_duplicate_protected_tags_spec.rb API: Protected tags 2018-08-25 05:38:54 +00:00
enqueue_delete_diff_files_workers_spec.rb Use schedulers and delete diff files upon deadtuples check 2018-07-10 10:25:48 -03:00
enqueue_redact_links_spec.rb Redact unsubscribe links in issuable texts 2018-10-23 21:20:20 +02:00
enqueue_verify_pages_domain_workers_spec.rb
fill_empty_finished_at_in_deployments_spec.rb Squashed commit of the following: 2018-11-06 10:54:27 +09:00
fill_file_store_spec.rb Add spec for fill_file_store migration 2018-05-28 14:50:10 +09:00
fix_wrongly_renamed_routes_spec.rb
generate_missing_routes_spec.rb Remove code for dynamically generating routes 2018-07-25 16:41:29 +02:00
import_common_metrics_spec.rb Fix migrations 2018-09-05 23:40:54 +02:00
issues_moved_to_id_foreign_key_spec.rb Updates from rubocop -a 2018-07-09 21:13:08 +08:00
migrate_build_stage_reference_again_spec.rb
migrate_cluster_configure_worker_sidekiq_queue_spec.rb Rename ClusterPlatformConfigureWorker 2018-12-21 13:17:13 +00:00
migrate_create_trace_artifact_sidekiq_queue_spec.rb
migrate_forbidden_redirect_uris_spec.rb Merge branch 'security-fix-uri-xss-applications' into 'master' 2018-11-28 19:14:15 -05:00
migrate_gcp_clusters_to_new_clusters_architectures_spec.rb Resolve Naming/UncommunicativeMethod 2018-07-09 21:13:08 +08:00
migrate_import_attributes_data_from_projects_to_project_mirror_data_spec.rb
migrate_issues_to_ghost_user_spec.rb Make it only run the associated migration 2018-11-27 22:48:55 +01:00
migrate_kubernetes_service_to_new_clusters_architectures_spec.rb
migrate_legacy_artifacts_to_job_artifacts_spec.rb Squashed commit of the following: 2018-08-27 09:12:02 +02:00
migrate_null_wiki_access_levels_spec.rb Add WikiAccessLevel migration from NULL to 20 2018-08-16 19:58:17 +02:00
migrate_object_storage_upload_sidekiq_queue_spec.rb Migrate jobs in object_storage_upload queue 2018-06-05 20:37:01 +02:00
migrate_old_artifacts_spec.rb Fix typos in comments and specs 2018-11-01 08:59:20 +02:00
migrate_pipeline_sidekiq_queues_spec.rb
migrate_pipeline_stages_spec.rb
migrate_process_commit_worker_jobs_spec.rb Remove direct disk access in RemoveDotGitFromUsernames 2018-07-19 15:19:45 +02:00
migrate_remaining_mr_metrics_populating_background_migration_spec.rb Take two for MR metrics population background migration 2018-05-30 11:51:29 -03:00
migrate_stage_id_reference_in_background_spec.rb Replace 'Sidekiq::Testing.inline!' with 'perform_enqueued_jobs' 2018-07-23 15:34:54 +11:00
migrate_stages_statuses_spec.rb Replace 'Sidekiq::Testing.inline!' with 'perform_enqueued_jobs' 2018-07-23 15:34:54 +11:00
migrate_update_head_pipeline_for_merge_request_sidekiq_queue_spec.rb
migrate_user_activities_to_users_last_activity_on_spec.rb
migrate_user_authentication_token_to_personal_access_token_spec.rb
migrate_user_project_view_spec.rb
move_personal_snippets_files_spec.rb
move_system_upload_folder_spec.rb
move_uploads_to_system_dir_spec.rb
normalize_ldap_extern_uids_spec.rb Fix interval in NormalizeLdapExternUids spec 2018-08-06 15:20:37 +02:00
populate_can_push_from_deploy_keys_projects_spec.rb
populate_mr_metrics_with_events_data_spec.rb Populate MR metrics with events table information (migration) 2018-12-06 22:08:38 -02:00
README.md
remove_assignee_id_from_issue_spec.rb
remove_dot_git_from_usernames_spec.rb
remove_duplicate_mr_events_spec.rb
remove_empty_extern_uid_auth0_identities_spec.rb
remove_empty_fork_networks_spec.rb
remove_orphaned_label_links_spec.rb Remove orphaned label links 2018-09-07 07:38:44 -07:00
remove_project_labels_group_id_spec.rb
remove_redundant_pipeline_stages_spec.rb
remove_soft_removed_objects_spec.rb Bring CE-EE parity to migrations/remove_soft_removed_objects_spec.rb 2018-06-18 12:18:02 -05:00
rename_duplicated_variable_key_spec.rb
rename_more_reserved_project_names_spec.rb Move Project#rename_repo to a service class 2018-10-22 15:12:46 +02:00
rename_reserved_project_names_spec.rb Move Project#rename_repo to a service class 2018-10-22 15:12:46 +02:00
rename_users_with_renamed_namespace_spec.rb
reschedule_builds_stages_migration_spec.rb
reschedule_commits_count_for_merge_request_diff_spec.rb
schedule_create_gpg_key_subkeys_from_gpg_keys_spec.rb Whitelist existing destroy_all offenses 2018-08-16 17:29:37 +02:00
schedule_digest_personal_access_tokens_spec.rb [master] Persist only SHA digest of PersonalAccessToken#token 2018-10-29 16:06:45 +00:00
schedule_merge_request_diff_migrations_spec.rb Replace 'Sidekiq::Testing.inline!' with 'perform_enqueued_jobs' 2018-07-23 15:34:54 +11:00
schedule_merge_request_diff_migrations_take_two_spec.rb Replace 'Sidekiq::Testing.inline!' with 'perform_enqueued_jobs' 2018-07-23 15:34:54 +11:00
schedule_merge_request_latest_merge_request_diff_id_migrations_spec.rb Replace 'Sidekiq::Testing.inline!' with 'perform_enqueued_jobs' 2018-07-23 15:34:54 +11:00
schedule_populate_merge_request_metrics_with_events_data_spec.rb
schedule_runners_token_encryption_spec.rb Adjust batch size for migrating runners token 2018-11-23 13:37:32 +01:00
schedule_set_confidential_note_events_on_webhooks_spec.rb Replace 'Sidekiq::Testing.inline!' with 'perform_enqueued_jobs' 2018-07-23 15:34:54 +11:00
schedule_stages_index_migration_spec.rb
schedule_to_archive_legacy_traces_spec.rb Fix specs. Rename migration file name which was conflicted with background migration's. 2018-06-04 14:14:20 +09:00
steal_fill_store_upload_spec.rb Add migration to steal FillStoreUpload 2018-11-07 11:29:31 +01:00
track_untracked_uploads_spec.rb
turn_nested_groups_into_regular_groups_for_mysql_spec.rb Deny repository disk access in development and test 2018-06-14 11:18:25 +00:00
update_legacy_diff_notes_type_for_import_spec.rb
update_notes_type_for_import_spec.rb
update_retried_for_ci_build_spec.rb
update_upload_paths_to_system_spec.rb

Testing migrations

In order to reliably test a migration, we need to test it against a database schema that this migration has been written for. In order to achieve that we have some migration helpers and RSpec test tag, called :migration.

If you want to write a test for a migration consider adding :migration tag to the test signature, like describe SomeMigrationClass, :migration.

How does it work?

Adding a :migration tag to a test signature injects a few before / after hooks to the test.

The most important change is that adding a :migration tag adds a before hook that will revert all migrations to the point that a migration under test is not yet migrated.

In other words, our custom RSpec hooks will find a previous migration, and migrate the database down to the previous migration version.

With this approach you can test a migration against a database schema that this migration has been written for.

Use migrate! helper to run the migration that is under test.

The after hook will migrate the database up and reinstitutes the latest schema version, so that the process does not affect subsequent specs and ensures proper isolation.

Testing a class that is not an ActiveRecord::Migration

In order to test a class that is not a migration itself, you will need to manually provide a required schema version. Please add a schema tag to a context that you want to switch the database schema within.

Example: describe SomeClass, :migration, schema: 20170608152748.

Available helpers

Use table helper to create a temporary ActiveRecord::Base derived model for a table.

Use migrate! helper to run the migration that is under test. It will not only run migration, but will also bump the schema version in the schema_migrations table. It is necessary because in the after hook we trigger the rest of the migrations, and we need to know where to start.

See spec/support/migrations_helpers.rb for all the available helpers.

An example

require 'spec_helper'

# Load a migration class.

require Rails.root.join('db', 'post_migrate', '20170526185842_migrate_pipeline_stages.rb')

describe MigratePipelineStages, :migration do

  # Create test data - pipeline and CI/CD jobs.

  let(:jobs) { table(:ci_builds) }
  let(:stages) { table(:ci_stages) }
  let(:pipelines) { table(:ci_pipelines) }
  let(:projects) { table(:projects) }

  before do
    projects.create!(id: 123, name: 'gitlab1', path: 'gitlab1')
    pipelines.create!(id: 1, project_id: 123, ref: 'master', sha: 'adf43c3a')
    jobs.create!(id: 1, commit_id: 1, project_id: 123, stage_idx: 2, stage: 'build')
    jobs.create!(id: 2, commit_id: 1, project_id: 123, stage_idx: 1, stage: 'test')
  end

  # Test the migration.

  it 'correctly migrates pipeline stages' do
    expect(stages.count).to be_zero

    migrate!

    expect(stages.count).to eq 2
    expect(stages.all.pluck(:name)).to match_array %w[test build]
  end
end

Best practices

  1. Note that this type of tests do not run within the transaction, we use a deletion database cleanup strategy. Do not depend on transaction being present.