From f7ae84f96fcc43c1857bf5d6bc6fb33a689852f0 Mon Sep 17 00:00:00 2001 From: Michael Kozono Date: Fri, 2 Nov 2018 15:59:09 -0700 Subject: [PATCH 1/4] Document limitation of bare repository import --- doc/raketasks/import.md | 47 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md index 97e9b36d1a6..05f2c9bd83d 100644 --- a/doc/raketasks/import.md +++ b/doc/raketasks/import.md @@ -6,6 +6,7 @@ - The groups will be created as needed, including subgroups - The owner of the group will be the first admin - Existing projects will be skipped +- Projects in hashed storage may be skipped (see [Importing bare repositories from hashed storage](#importing-bare-repositories-from-hashed-storage)) - The existing Git repos will be moved from disk (removed from the original path) ## How to use @@ -26,7 +27,6 @@ sudo -u git mkdir /var/opt/gitlab/git-data/repository-import-/new_group If we copy the repos to `/var/opt/gitlab/git-data/repository-import-`, and repo A needs to be under the groups G1 and G2, it will have to be created under those folders: `/var/opt/gitlab/git-data/repository-import-/G1/G2/A.git`. - ``` sudo cp -r /old/git/foo.git /var/opt/gitlab/git-data/repository-import-/new_group/ @@ -70,3 +70,48 @@ Processing /var/opt/gitlab/git-data/repository-import-1/group/xyz.git * Skipping repo /var/opt/gitlab/git-data/repository-import-1/@shared/a/b/abcd.git [...] ``` + +## Importing bare repositories from hashed storage + +### Background + +Projects in legacy storage have a directory structure that mirrors their full +project path in GitLab, including their namespace structure. This information is +leveraged by the bare repository importer to import projects into their proper +locations. Each project and its parent namespaces are meaningfully named. + +However, the directory structure of projects in hashed storage do not contain +this information. This is beneficial for a variety of reasons, especially +improved performance and data integrity. See +[Repository Storage Types](../administration/repository_storage_types.md) for +more details. + +### Which repositories are importable? + +#### v10.3 or earlier + +Importing bare repositories from hashed storage is unsupported. + +#### v10.4 and later + +In order to support this, we began storing the full GitLab project path with +each repository. However, existing repositories were not migrated to include +this path. + +The following are importable as bare repositories: + +- Created in hashed storage in v10.4+ +- Migrated to hashed storage in v10.4+ +- Renamed in v10.4+ +- Transferred to another namespace in v10.4+ +- Ancestor renamed in v10.4+ +- Ancestor transferred to another namespace in v10.4+ + +The following are **not** importable as bare repositories: + +- Created in or migrated to hashed storage in v10.3 or earlier, and was not + renamed or transferred in v10.4+, and whose ancestor namespaces were not + renamed or transferred in v10.4+. + +There is an [open issue to add a migration to make all bare repositories +importable](https://gitlab.com/gitlab-org/gitlab-ce/issues/41776). \ No newline at end of file From d65f9cdbaa7607c009e8dddee15e4685a789ed30 Mon Sep 17 00:00:00 2001 From: Michael Kozono Date: Wed, 7 Nov 2018 12:32:23 -0800 Subject: [PATCH 2/4] Add steps to manually migrate projects --- doc/raketasks/import.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md index 05f2c9bd83d..a1a6cfcce7f 100644 --- a/doc/raketasks/import.md +++ b/doc/raketasks/import.md @@ -114,4 +114,21 @@ The following are **not** importable as bare repositories: renamed or transferred in v10.4+. There is an [open issue to add a migration to make all bare repositories -importable](https://gitlab.com/gitlab-org/gitlab-ce/issues/41776). \ No newline at end of file +importable](https://gitlab.com/gitlab-org/gitlab-ce/issues/41776). + +Until then, you may wish to manually migrate repositories yourself. You can use +[Rails console](https://docs.gitlab.com/omnibus/maintenance/#starting-a-rails-console-session) +to do so. In a Rails console session, run the following to migrate a project: + +``` +project = Project.find_by_full_path('gitlab-org/gitlab-ce') +project.write_repository_config +``` + +In a Rails console session, run the following to migrate all of a namespace's +projects (this may take awhile if there are 1000s of projects in a namespace): + +``` +namespace = Namespace.find_by_full_path('gitlab-org') +namespace.send(:write_projects_repository_config) +``` \ No newline at end of file From 36c693575bb4f21fc01dc49add68505e80643b86 Mon Sep 17 00:00:00 2001 From: Michael Kozono Date: Wed, 7 Nov 2018 12:40:07 -0800 Subject: [PATCH 3/4] Add an example of the stored project path --- doc/raketasks/import.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md index a1a6cfcce7f..dc7435e76d1 100644 --- a/doc/raketasks/import.md +++ b/doc/raketasks/import.md @@ -95,8 +95,15 @@ Importing bare repositories from hashed storage is unsupported. #### v10.4 and later In order to support this, we began storing the full GitLab project path with -each repository. However, existing repositories were not migrated to include -this path. +each repository, in a special section of the git repository's config file. This +section is formatted as follows: + +``` +[gitlab] + fullpath = gitlab-org/gitlab-ce +``` + +However, existing repositories were not migrated to include this path. The following are importable as bare repositories: From 33f1ced6b868032bceb0cbe148f1dc0f057d5729 Mon Sep 17 00:00:00 2001 From: Michael Kozono Date: Wed, 7 Nov 2018 12:52:18 -0800 Subject: [PATCH 4/4] Improve wording and consistency --- doc/raketasks/import.md | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md index dc7435e76d1..bb316df5b9a 100644 --- a/doc/raketasks/import.md +++ b/doc/raketasks/import.md @@ -88,15 +88,15 @@ more details. ### Which repositories are importable? -#### v10.3 or earlier +#### GitLab 10.3 or earlier Importing bare repositories from hashed storage is unsupported. -#### v10.4 and later +#### GitLab 10.4 and later -In order to support this, we began storing the full GitLab project path with -each repository, in a special section of the git repository's config file. This -section is formatted as follows: +To support importing bare repositories from hashed storage, GitLab 10.4 and +later stores the full project path with each repository, in a special section of +the git repository's config file. This section is formatted as follows: ``` [gitlab] @@ -105,20 +105,21 @@ section is formatted as follows: However, existing repositories were not migrated to include this path. -The following are importable as bare repositories: +Bare repositories are importable if the following events occurred to the +repository in GitLab 10.4 and later: -- Created in hashed storage in v10.4+ -- Migrated to hashed storage in v10.4+ -- Renamed in v10.4+ -- Transferred to another namespace in v10.4+ -- Ancestor renamed in v10.4+ -- Ancestor transferred to another namespace in v10.4+ +- Created +- Migrated to hashed storage +- Renamed +- Transferred to another namespace +- Ancestor renamed +- Ancestor transferred to another namespace -The following are **not** importable as bare repositories: +Bare repositories are **not** importable by GitLab 10.4 and later when all the following are true about the repository: -- Created in or migrated to hashed storage in v10.3 or earlier, and was not - renamed or transferred in v10.4+, and whose ancestor namespaces were not - renamed or transferred in v10.4+. +- It was created in GitLab 10.3 or earlier. +- It was not renamed, transferred, or migrated to hashed storage in GitLab 10.4 and later. +- Its ancestor namespaces were not renamed or transferred in GitLab 10.4 and later. There is an [open issue to add a migration to make all bare repositories importable](https://gitlab.com/gitlab-org/gitlab-ce/issues/41776). @@ -133,9 +134,9 @@ project.write_repository_config ``` In a Rails console session, run the following to migrate all of a namespace's -projects (this may take awhile if there are 1000s of projects in a namespace): +projects (this may take a while if there are 1000s of projects in a namespace): ``` namespace = Namespace.find_by_full_path('gitlab-org') namespace.send(:write_projects_repository_config) -``` \ No newline at end of file +```