gitlab-org--gitlab-foss/spec/migrations
Kamil Trzciński 0103d5be96 Add config_options|variables to BuildMetadata
These are data columns that store runtime configuration
of build needed to execute it on runner and within pipeline.

The definition of this data is that once used, and when no longer
needed (due to retry capability) they can be freely removed.

They use `jsonb` on PostgreSQL, and `text` on MySQL (due to lacking
support for json datatype on old enough version).
2019-01-04 16:38:17 +01:00
..
active_record
README.md
add_foreign_key_from_notification_settings_to_users_spec.rb
add_foreign_key_to_merge_requests_spec.rb
add_foreign_keys_to_todos_spec.rb
add_head_pipeline_for_each_merge_request_spec.rb
add_not_null_constraint_to_project_mirror_data_foreign_key_spec.rb
add_pages_access_level_to_project_feature_spec.rb
add_pipeline_build_foreign_key_spec.rb
add_unique_constraint_to_project_features_project_id_spec.rb
assure_commits_count_for_merge_request_diff_spec.rb
backfill_releases_name_with_tag_name_spec.rb
backfill_store_project_full_path_in_repo_spec.rb
calculate_conv_dev_index_percentages_spec.rb
change_default_value_for_dsa_key_restriction_spec.rb
clean_appearance_symlinks_spec.rb
clean_stage_id_reference_migration_spec.rb
clean_stages_statuses_migration_spec.rb
clean_up_for_members_spec.rb
clean_upload_symlinks_spec.rb
cleanup_build_stage_migration_spec.rb
cleanup_environments_external_url_spec.rb
cleanup_move_system_upload_folder_symlink_spec.rb
cleanup_namespaceless_pending_delete_projects_spec.rb
cleanup_nonexisting_namespace_pending_delete_projects_spec.rb
cleanup_stages_position_migration_spec.rb
convert_custom_notification_settings_to_columns_spec.rb
create_missing_namespace_for_internal_users_spec.rb
delete_conflicting_redirect_routes_spec.rb
delete_inconsistent_internal_id_records_spec.rb Add config_options|variables to BuildMetadata 2019-01-04 16:38:17 +01:00
drop_duplicate_protected_tags_spec.rb
enqueue_delete_diff_files_workers_spec.rb
enqueue_redact_links_spec.rb
enqueue_verify_pages_domain_workers_spec.rb
fill_empty_finished_at_in_deployments_spec.rb
fill_file_store_spec.rb
fix_wrongly_renamed_routes_spec.rb
generate_missing_routes_spec.rb
import_common_metrics_spec.rb
issues_moved_to_id_foreign_key_spec.rb
migrate_build_stage_reference_again_spec.rb
migrate_cluster_configure_worker_sidekiq_queue_spec.rb
migrate_create_trace_artifact_sidekiq_queue_spec.rb
migrate_forbidden_redirect_uris_spec.rb
migrate_gcp_clusters_to_new_clusters_architectures_spec.rb
migrate_import_attributes_data_from_projects_to_project_mirror_data_spec.rb
migrate_issues_to_ghost_user_spec.rb
migrate_kubernetes_service_to_new_clusters_architectures_spec.rb
migrate_legacy_artifacts_to_job_artifacts_spec.rb
migrate_null_wiki_access_levels_spec.rb
migrate_object_storage_upload_sidekiq_queue_spec.rb
migrate_old_artifacts_spec.rb
migrate_pipeline_sidekiq_queues_spec.rb
migrate_pipeline_stages_spec.rb
migrate_process_commit_worker_jobs_spec.rb
migrate_remaining_mr_metrics_populating_background_migration_spec.rb
migrate_stage_id_reference_in_background_spec.rb
migrate_stages_statuses_spec.rb
migrate_update_head_pipeline_for_merge_request_sidekiq_queue_spec.rb
migrate_user_activities_to_users_last_activity_on_spec.rb
migrate_user_authentication_token_to_personal_access_token_spec.rb
migrate_user_project_view_spec.rb
move_personal_snippets_files_spec.rb
move_system_upload_folder_spec.rb
move_uploads_to_system_dir_spec.rb
normalize_ldap_extern_uids_spec.rb
populate_can_push_from_deploy_keys_projects_spec.rb
populate_mr_metrics_with_events_data_spec.rb
remove_assignee_id_from_issue_spec.rb
remove_dot_git_from_usernames_spec.rb
remove_duplicate_mr_events_spec.rb
remove_empty_extern_uid_auth0_identities_spec.rb
remove_empty_fork_networks_spec.rb
remove_orphaned_label_links_spec.rb
remove_project_labels_group_id_spec.rb
remove_redundant_pipeline_stages_spec.rb
remove_soft_removed_objects_spec.rb
rename_duplicated_variable_key_spec.rb
rename_more_reserved_project_names_spec.rb
rename_reserved_project_names_spec.rb
rename_users_with_renamed_namespace_spec.rb
reschedule_builds_stages_migration_spec.rb
reschedule_commits_count_for_merge_request_diff_spec.rb
schedule_create_gpg_key_subkeys_from_gpg_keys_spec.rb
schedule_digest_personal_access_tokens_spec.rb
schedule_merge_request_diff_migrations_spec.rb
schedule_merge_request_diff_migrations_take_two_spec.rb
schedule_merge_request_latest_merge_request_diff_id_migrations_spec.rb
schedule_populate_merge_request_metrics_with_events_data_spec.rb
schedule_runners_token_encryption_spec.rb
schedule_set_confidential_note_events_on_webhooks_spec.rb
schedule_stages_index_migration_spec.rb
schedule_to_archive_legacy_traces_spec.rb
steal_fill_store_upload_spec.rb
track_untracked_uploads_spec.rb
turn_nested_groups_into_regular_groups_for_mysql_spec.rb
update_legacy_diff_notes_type_for_import_spec.rb
update_notes_type_for_import_spec.rb
update_retried_for_ci_build_spec.rb
update_upload_paths_to_system_spec.rb

README.md

Testing migrations

In order to reliably test a migration, we need to test it against a database schema that this migration has been written for. In order to achieve that we have some migration helpers and RSpec test tag, called :migration.

If you want to write a test for a migration consider adding :migration tag to the test signature, like describe SomeMigrationClass, :migration.

How does it work?

Adding a :migration tag to a test signature injects a few before / after hooks to the test.

The most important change is that adding a :migration tag adds a before hook that will revert all migrations to the point that a migration under test is not yet migrated.

In other words, our custom RSpec hooks will find a previous migration, and migrate the database down to the previous migration version.

With this approach you can test a migration against a database schema that this migration has been written for.

Use migrate! helper to run the migration that is under test.

The after hook will migrate the database up and reinstitutes the latest schema version, so that the process does not affect subsequent specs and ensures proper isolation.

Testing a class that is not an ActiveRecord::Migration

In order to test a class that is not a migration itself, you will need to manually provide a required schema version. Please add a schema tag to a context that you want to switch the database schema within.

Example: describe SomeClass, :migration, schema: 20170608152748.

Available helpers

Use table helper to create a temporary ActiveRecord::Base derived model for a table.

Use migrate! helper to run the migration that is under test. It will not only run migration, but will also bump the schema version in the schema_migrations table. It is necessary because in the after hook we trigger the rest of the migrations, and we need to know where to start.

See spec/support/migrations_helpers.rb for all the available helpers.

An example

require 'spec_helper'

# Load a migration class.

require Rails.root.join('db', 'post_migrate', '20170526185842_migrate_pipeline_stages.rb')

describe MigratePipelineStages, :migration do

  # Create test data - pipeline and CI/CD jobs.

  let(:jobs) { table(:ci_builds) }
  let(:stages) { table(:ci_stages) }
  let(:pipelines) { table(:ci_pipelines) }
  let(:projects) { table(:projects) }

  before do
    projects.create!(id: 123, name: 'gitlab1', path: 'gitlab1')
    pipelines.create!(id: 1, project_id: 123, ref: 'master', sha: 'adf43c3a')
    jobs.create!(id: 1, commit_id: 1, project_id: 123, stage_idx: 2, stage: 'build')
    jobs.create!(id: 2, commit_id: 1, project_id: 123, stage_idx: 1, stage: 'test')
  end

  # Test the migration.

  it 'correctly migrates pipeline stages' do
    expect(stages.count).to be_zero

    migrate!

    expect(stages.count).to eq 2
    expect(stages.all.pluck(:name)).to match_array %w[test build]
  end
end

Best practices

  1. Note that this type of tests do not run within the transaction, we use a deletion database cleanup strategy. Do not depend on transaction being present.