gitlab-org--gitlab-foss/spec/migrations
Jan Provaznik 7b22381603 Reschedule commits_count background migration
We still have >100K unmigrated MergeRequestDiffs
which don't have commits_count set yet (see
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17567#note_61904891)

This migration re-schedules the original background migration.
To assure that records are not processed twice, records with
commits_count set are skipped.

Related to #41698 and !17567
2018-03-09 09:56:49 +01:00
..
active_record Enable RSpec/FilePath cop 2017-04-26 12:50:32 +01:00
add_foreign_key_to_merge_requests_spec.rb Use batching to clear orphans in head_pipeline migration 2017-07-18 12:07:12 +02:00
add_foreign_keys_to_todos_spec.rb Add foreign keys to todos table. 2018-02-05 16:16:57 +01:00
add_head_pipeline_for_each_merge_request_spec.rb Use the DatabaseCleaner 'deletion' strategy instead of 'truncation' 2018-01-24 17:25:55 +00:00
calculate_conv_dev_index_percentages_spec.rb Use the DatabaseCleaner 'deletion' strategy instead of 'truncation' 2018-01-24 17:25:55 +00:00
clean_appearance_symlinks_spec.rb Merge branch 'bvl-remove-appearance-symlink' into 'security-9-3' 2017-07-19 22:28:17 -05:00
clean_stage_id_reference_migration_spec.rb Fix background migration cleanup specs 2017-07-20 10:53:52 +02:00
clean_stages_statuses_migration_spec.rb Make sure all stages are migrated after a cleanup 2017-09-18 13:47:46 +02:00
clean_up_for_members_spec.rb Fix user membership destroy relation 2018-01-02 15:06:44 +00:00
clean_upload_symlinks_spec.rb Update migrations to move directly into the -/system folder 2017-08-11 18:15:02 +02:00
cleanup_move_system_upload_folder_symlink_spec.rb Move the uploads/system folder to uploads/-/system 2017-07-18 15:38:54 +02:00
cleanup_namespaceless_pending_delete_projects_spec.rb Update specs to respect Project#namespace_id not null constraint 2018-03-01 16:09:09 +02:00
cleanup_nonexisting_namespace_pending_delete_projects_spec.rb Migration to remove pending delete projects with non-existing namespace 2017-08-22 14:29:54 +02:00
convert_custom_notification_settings_to_columns_spec.rb Reset column information after the schema is migrated in MigrationsHelpers.schema_migrate_up! 2018-02-07 11:38:50 +01:00
delete_conflicting_redirect_routes_spec.rb Make DeleteConflictingRedirectRoutes no-op 2018-01-03 12:23:20 -08:00
enqueue_verify_pages_domain_workers_spec.rb Add DNS verification to Pages custom domains 2018-02-23 12:22:29 +00:00
fix_wrongly_renamed_routes_spec.rb Use the DatabaseCleaner 'deletion' strategy instead of 'truncation' 2018-01-24 17:25:55 +00:00
issues_moved_to_id_foreign_key_spec.rb Fix PostgreSQL implementation of migration 2017-12-27 04:12:18 -08:00
migrate_build_stage_reference_again_spec.rb Rename stage ref migration specs to match a class name 2017-06-21 15:51:00 +02:00
migrate_create_trace_artifact_sidekiq_queue_spec.rb Fix tests by latest proposal changes 2018-03-06 21:44:15 +09:00
migrate_gcp_clusters_to_new_clusters_architectures_spec.rb Prevent cluster migration spec from interfering with later specs 2018-01-05 17:44:38 +00:00
migrate_issues_to_ghost_user_spec.rb Update specs to respect Project#namespace_id not null constraint 2018-03-01 16:09:09 +02:00
migrate_kubernetes_service_to_new_clusters_architectures_spec.rb Fix static analysys 2018-01-08 17:01:29 +09:00
migrate_old_artifacts_spec.rb Fix specs failures, and use factory with :ci_job_artifact, :archive 2017-12-03 12:04:49 +01:00
migrate_pipeline_sidekiq_queues_spec.rb Migrate new jobs when reverting pipeline queues migration 2017-08-25 14:25:18 +02:00
migrate_pipeline_stages_spec.rb Migrate stages only with correct foreign references 2017-06-06 14:32:24 +02:00
migrate_process_commit_worker_jobs_spec.rb Use hashed storage in the specs 2018-02-07 13:40:46 +00:00
migrate_stage_id_reference_in_background_spec.rb Refactor matchers for background migrations 2018-01-05 13:40:31 +01:00
migrate_stages_statuses_spec.rb Tag migration specs with :sidekiq to clear queues between tests 2018-02-28 15:35:43 -08:00
migrate_update_head_pipeline_for_merge_request_sidekiq_queue_spec.rb Move update_head_pipeline_for_merge_request queue to pipeline_processing namespace 2018-03-07 01:39:00 +09:00
migrate_user_activities_to_users_last_activity_on_spec.rb Use the DatabaseCleaner 'deletion' strategy instead of 'truncation' 2018-01-24 17:25:55 +00:00
migrate_user_authentication_token_to_personal_access_token_spec.rb Migrate user private tokens to personal access tokens 2017-11-02 11:39:03 +01:00
migrate_user_project_view_spec.rb Use the DatabaseCleaner 'deletion' strategy instead of 'truncation' 2018-01-24 17:25:55 +00:00
move_personal_snippets_files_spec.rb Move the personal snippet uploads from system to -/system 2017-08-11 18:17:45 +02:00
move_system_upload_folder_spec.rb Update migrations to move directly into the -/system folder 2017-08-11 18:15:02 +02:00
move_uploads_to_system_dir_spec.rb Update migrations to move directly into the -/system folder 2017-08-11 18:15:02 +02:00
normalize_ldap_extern_uids_spec.rb Run background migrations with a minimum interval 2018-01-05 16:23:25 +01:00
populate_can_push_from_deploy_keys_projects_spec.rb Applied fix 2018-01-17 21:42:18 +01:00
README.md Avoid dropping tables in test 2018-02-14 12:31:16 +01:00
remove_assignee_id_from_issue_spec.rb Add spec for removing issues.assignee_id 2017-12-11 14:09:55 +00:00
remove_dot_git_from_usernames_spec.rb Fix spec 2017-08-17 14:39:02 -07:00
remove_duplicate_mr_events_spec.rb Use the DatabaseCleaner 'deletion' strategy instead of 'truncation' 2018-01-24 17:25:55 +00:00
remove_empty_fork_networks_spec.rb Fix the Upload model being tainted after migration spec 2018-01-22 15:31:41 +00:00
remove_project_labels_group_id_spec.rb fix spec 2018-02-02 06:36:03 -08:00
remove_redundant_pipeline_stages_spec.rb Add more specs for unique stages index migration 2018-02-06 09:58:50 +01:00
remove_soft_removed_objects_spec.rb Added tests for removing soft removed objects 2018-01-10 17:33:03 +01:00
rename_duplicated_variable_key_spec.rb Rename the migration test to match the migration path 2017-06-27 18:29:23 +08:00
rename_more_reserved_project_names_spec.rb Use the DatabaseCleaner 'deletion' strategy instead of 'truncation' 2018-01-24 17:25:55 +00:00
rename_reserved_project_names_spec.rb Use the latest migration in spec/migrations/rename_reserved_project_names_spec.rb 2018-02-12 16:54:22 +01:00
rename_users_with_renamed_namespace_spec.rb Use the DatabaseCleaner 'deletion' strategy instead of 'truncation' 2018-01-24 17:25:55 +00:00
reschedule_commits_count_for_merge_request_diff_spec.rb Reschedule commits_count background migration 2018-03-09 09:56:49 +01:00
schedule_build_stage_migration_spec.rb Tag migration specs with :sidekiq to clear queues between tests 2018-02-28 15:35:43 -08:00
schedule_create_gpg_key_subkeys_from_gpg_keys_spec.rb Refactor matchers for background migrations 2018-01-05 13:40:31 +01:00
schedule_merge_request_diff_migrations_spec.rb Refactor matchers for background migrations 2018-01-05 13:40:31 +01:00
schedule_merge_request_diff_migrations_take_two_spec.rb Refactor matchers for background migrations 2018-01-05 13:40:31 +01:00
schedule_merge_request_latest_merge_request_diff_id_migrations_spec.rb Refactor matchers for background migrations 2018-01-05 13:40:31 +01:00
schedule_populate_merge_request_metrics_with_events_data_spec.rb Denormalize commits count for merge request diffs 2018-01-10 20:40:02 +01:00
track_untracked_uploads_spec.rb Move TrackUntrackedUploadsHelpers 2018-02-20 12:09:50 -08:00
turn_nested_groups_into_regular_groups_for_mysql_spec.rb Use hashed storage in the specs 2018-02-07 13:40:46 +00:00
update_legacy_diff_notes_type_for_import_spec.rb Fix notes type created from import 2017-09-29 11:18:58 +00:00
update_notes_type_for_import_spec.rb Fix notes type created from import 2017-09-29 11:18:58 +00:00
update_retried_for_ci_build_spec.rb Use the DatabaseCleaner 'deletion' strategy instead of 'truncation' 2018-01-24 17:25:55 +00:00
update_upload_paths_to_system_spec.rb Fix spec/migrations/update_upload_paths_to_system_spec.rb to use the :migration metadata 2018-01-18 12:17:22 +01:00

Testing migrations

In order to reliably test a migration, we need to test it against a database schema that this migration has been written for. In order to achieve that we have some migration helpers and RSpec test tag, called :migration.

If you want to write a test for a migration consider adding :migration tag to the test signature, like describe SomeMigrationClass, :migration.

How does it work?

Adding a :migration tag to a test signature injects a few before / after hooks to the test.

The most important change is that adding a :migration tag adds a before hook that will revert all migrations to the point that a migration under test is not yet migrated.

In other words, our custom RSpec hooks will find a previous migration, and migrate the database down to the previous migration version.

With this approach you can test a migration against a database schema that this migration has been written for.

Use migrate! helper to run the migration that is under test.

The after hook will migrate the database up and reinstitutes the latest schema version, so that the process does not affect subsequent specs and ensures proper isolation.

Testing a class that is not an ActiveRecord::Migration

In order to test a class that is not a migration itself, you will need to manually provide a required schema version. Please add a schema tag to a context that you want to switch the database schema within.

Example: describe SomeClass, :migration, schema: 20170608152748.

Available helpers

Use table helper to create a temporary ActiveRecord::Base derived model for a table.

Use migrate! helper to run the migration that is under test. It will not only run migration, but will also bump the schema version in the schema_migrations table. It is necessary because in the after hook we trigger the rest of the migrations, and we need to know where to start.

See spec/support/migrations_helpers.rb for all the available helpers.

An example

require 'spec_helper'

# Load a migration class.

require Rails.root.join('db', 'post_migrate', '20170526185842_migrate_pipeline_stages.rb')

describe MigratePipelineStages, :migration do

  # Create test data - pipeline and CI/CD jobs.

  let(:jobs) { table(:ci_builds) }
  let(:stages) { table(:ci_stages) }
  let(:pipelines) { table(:ci_pipelines) }
  let(:projects) { table(:projects) }

  before do
    projects.create!(id: 123, name: 'gitlab1', path: 'gitlab1')
    pipelines.create!(id: 1, project_id: 123, ref: 'master', sha: 'adf43c3a')
    jobs.create!(id: 1, commit_id: 1, project_id: 123, stage_idx: 2, stage: 'build')
    jobs.create!(id: 2, commit_id: 1, project_id: 123, stage_idx: 1, stage: 'test')
  end

  # Test the migration.

  it 'correctly migrates pipeline stages' do
    expect(stages.count).to be_zero

    migrate!

    expect(stages.count).to eq 2
    expect(stages.all.pluck(:name)).to match_array %w[test build]
  end
end

Best practices

  1. Note that this type of tests do not run within the transaction, we use a deletion database cleanup strategy. Do not depend on transaction being present.