gitlab-org--gitlab-foss/spec/migrations
Stan Hu 3126e89eb8
Add a unique and not null constraint on the project_features.project_id column
This commit has two migrations:

1. The first prunes duplicate rows in the project_features table and leaves
   the row with the highest ID.  Since the behavior was indeterministic before
   and depended on which row the database decided to use, this change at least
   makes the permissions consistent. For example, in some cases, the Wiki may
   have been disabled but enabled in another entry.

2. The second adds a non-null constraint on the project_features.project_id
   column.

Closes #37882

Fixes a significant part of gitlab-com/migration#408.

We found that we were overcounting Wikis because of these duplicates.
On GitLab.com, there are 56 rows with duplicate entries by project_id, and 16,661 rows with NULL project_id values.
2018-05-15 14:20:14 +02:00
..
active_record Merge branch 'jej/mattermost-notification-confidentiality-10-6' into 'security-10-6' 2018-04-05 08:41:56 +02:00
add_foreign_key_to_merge_requests_spec.rb
add_foreign_keys_to_todos_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
add_head_pipeline_for_each_merge_request_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
add_not_null_constraint_to_project_mirror_data_foreign_key_spec.rb Adds not null constraint to ProjectMirrorData's foreign_key project_id 2018-05-08 11:25:12 +01:00
add_pipeline_build_foreign_key_spec.rb Remove orphaned builds without project / pipeline reference 2018-05-08 17:12:29 +02:00
add_unique_constraint_to_project_features_project_id_spec.rb Add a unique and not null constraint on the project_features.project_id column 2018-05-15 14:20:14 +02:00
assure_commits_count_for_merge_request_diff_spec.rb Cleanup after adding MR diff's commit_count (try 2) 2018-04-25 10:53:37 +02:00
calculate_conv_dev_index_percentages_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
clean_appearance_symlinks_spec.rb
clean_stage_id_reference_migration_spec.rb
clean_stages_statuses_migration_spec.rb
clean_up_for_members_spec.rb
clean_upload_symlinks_spec.rb
cleanup_build_stage_migration_spec.rb Cleanup pipeline build stage background migration 2018-05-07 14:38:11 +02:00
cleanup_move_system_upload_folder_symlink_spec.rb
cleanup_namespaceless_pending_delete_projects_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
cleanup_nonexisting_namespace_pending_delete_projects_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
convert_custom_notification_settings_to_columns_spec.rb Reset column information after the schema is migrated in MigrationsHelpers.schema_migrate_up! 2018-02-07 11:38:50 +01:00
create_missing_namespace_for_internal_users_spec.rb Fix missing namespace for some internal users 2018-04-20 17:53:22 -05:00
delete_conflicting_redirect_routes_spec.rb
enqueue_verify_pages_domain_workers_spec.rb Add DNS verification to Pages custom domains 2018-02-23 12:22:29 +00:00
fix_wrongly_renamed_routes_spec.rb Use the DatabaseCleaner 'deletion' strategy instead of 'truncation' 2018-01-24 17:25:55 +00:00
issues_moved_to_id_foreign_key_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
migrate_build_stage_reference_again_spec.rb
migrate_create_trace_artifact_sidekiq_queue_spec.rb Fix tests by latest proposal changes 2018-03-06 21:44:15 +09:00
migrate_gcp_clusters_to_new_clusters_architectures_spec.rb Merge branch 'jej/mattermost-notification-confidentiality-10-6' into 'security-10-6' 2018-04-05 08:41:56 +02:00
migrate_import_attributes_data_from_projects_to_project_mirror_data_spec.rb Backports every CE related change from ee-44542 to CE 2018-05-04 17:33:26 +02:00
migrate_issues_to_ghost_user_spec.rb Update specs to respect Project#namespace_id not null constraint 2018-03-01 16:09:09 +02:00
migrate_kubernetes_service_to_new_clusters_architectures_spec.rb
migrate_old_artifacts_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
migrate_pipeline_sidekiq_queues_spec.rb
migrate_pipeline_stages_spec.rb
migrate_process_commit_worker_jobs_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
migrate_stage_id_reference_in_background_spec.rb
migrate_stages_statuses_spec.rb Tag migration specs with :sidekiq to clear queues between tests 2018-02-28 15:35:43 -08:00
migrate_update_head_pipeline_for_merge_request_sidekiq_queue_spec.rb Move update_head_pipeline_for_merge_request queue to pipeline_processing namespace 2018-03-07 01:39:00 +09:00
migrate_user_activities_to_users_last_activity_on_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
migrate_user_authentication_token_to_personal_access_token_spec.rb
migrate_user_project_view_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
move_personal_snippets_files_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
move_system_upload_folder_spec.rb
move_uploads_to_system_dir_spec.rb
normalize_ldap_extern_uids_spec.rb
populate_can_push_from_deploy_keys_projects_spec.rb
README.md Avoid dropping tables in test 2018-02-14 12:31:16 +01:00
remove_assignee_id_from_issue_spec.rb
remove_dot_git_from_usernames_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
remove_duplicate_mr_events_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
remove_empty_extern_uid_auth0_identities_spec.rb Merge branch 'fix/auth0-unsafe-login-10-6' into 'security-10-6' 2018-03-21 14:43:47 +00:00
remove_empty_fork_networks_spec.rb Merge commit '8af23def1d6' into object-storage-ee-to-ce-backport 2018-02-28 21:11:53 +01:00
remove_project_labels_group_id_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
remove_redundant_pipeline_stages_spec.rb Add more specs for unique stages index migration 2018-02-06 09:58:50 +01:00
remove_soft_removed_objects_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
rename_duplicated_variable_key_spec.rb
rename_more_reserved_project_names_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
rename_reserved_project_names_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
rename_users_with_renamed_namespace_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
reschedule_builds_stages_migration_spec.rb Update entities in pipeline stages migration test 2018-04-05 12:08:04 +02:00
reschedule_commits_count_for_merge_request_diff_spec.rb Reschedule commits_count background migration 2018-03-09 09:56:49 +01:00
schedule_create_gpg_key_subkeys_from_gpg_keys_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
schedule_merge_request_diff_migrations_spec.rb
schedule_merge_request_diff_migrations_take_two_spec.rb
schedule_merge_request_latest_merge_request_diff_id_migrations_spec.rb
schedule_populate_merge_request_metrics_with_events_data_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
schedule_set_confidential_note_events_on_webhooks_spec.rb Merge branch 'jej/mattermost-notification-confidentiality-10-6' into 'security-10-6' 2018-04-05 08:41:56 +02:00
schedule_stages_index_migration_spec.rb Use stages position column to track stage index 2018-05-01 14:30:44 +02:00
track_untracked_uploads_spec.rb Move TrackUntrackedUploadsHelpers 2018-02-20 12:09:50 -08:00
turn_nested_groups_into_regular_groups_for_mysql_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
update_legacy_diff_notes_type_for_import_spec.rb
update_notes_type_for_import_spec.rb
update_retried_for_ci_build_spec.rb Ignore current RSpec/FactoriesInMigrationSpecs offenses 2018-04-06 17:41:52 +02:00
update_upload_paths_to_system_spec.rb Fix spec/migrations/update_upload_paths_to_system_spec.rb to use the :migration metadata 2018-01-18 12:17:22 +01:00

Testing migrations

In order to reliably test a migration, we need to test it against a database schema that this migration has been written for. In order to achieve that we have some migration helpers and RSpec test tag, called :migration.

If you want to write a test for a migration consider adding :migration tag to the test signature, like describe SomeMigrationClass, :migration.

How does it work?

Adding a :migration tag to a test signature injects a few before / after hooks to the test.

The most important change is that adding a :migration tag adds a before hook that will revert all migrations to the point that a migration under test is not yet migrated.

In other words, our custom RSpec hooks will find a previous migration, and migrate the database down to the previous migration version.

With this approach you can test a migration against a database schema that this migration has been written for.

Use migrate! helper to run the migration that is under test.

The after hook will migrate the database up and reinstitutes the latest schema version, so that the process does not affect subsequent specs and ensures proper isolation.

Testing a class that is not an ActiveRecord::Migration

In order to test a class that is not a migration itself, you will need to manually provide a required schema version. Please add a schema tag to a context that you want to switch the database schema within.

Example: describe SomeClass, :migration, schema: 20170608152748.

Available helpers

Use table helper to create a temporary ActiveRecord::Base derived model for a table.

Use migrate! helper to run the migration that is under test. It will not only run migration, but will also bump the schema version in the schema_migrations table. It is necessary because in the after hook we trigger the rest of the migrations, and we need to know where to start.

See spec/support/migrations_helpers.rb for all the available helpers.

An example

require 'spec_helper'

# Load a migration class.

require Rails.root.join('db', 'post_migrate', '20170526185842_migrate_pipeline_stages.rb')

describe MigratePipelineStages, :migration do

  # Create test data - pipeline and CI/CD jobs.

  let(:jobs) { table(:ci_builds) }
  let(:stages) { table(:ci_stages) }
  let(:pipelines) { table(:ci_pipelines) }
  let(:projects) { table(:projects) }

  before do
    projects.create!(id: 123, name: 'gitlab1', path: 'gitlab1')
    pipelines.create!(id: 1, project_id: 123, ref: 'master', sha: 'adf43c3a')
    jobs.create!(id: 1, commit_id: 1, project_id: 123, stage_idx: 2, stage: 'build')
    jobs.create!(id: 2, commit_id: 1, project_id: 123, stage_idx: 1, stage: 'test')
  end

  # Test the migration.

  it 'correctly migrates pipeline stages' do
    expect(stages.count).to be_zero

    migrate!

    expect(stages.count).to eq 2
    expect(stages.all.pluck(:name)).to match_array %w[test build]
  end
end

Best practices

  1. Note that this type of tests do not run within the transaction, we use a deletion database cleanup strategy. Do not depend on transaction being present.