Commit graph

13 commits

Author SHA1 Message Date
Jacob Vosmaer
a02e22438d Fix wrong use of ActiveRecord in PoolRepository 2019-04-20 13:27:53 +00:00
Jacob Vosmaer
35b9274f12 Stop calling UnlinkRepositoryFromObjectPool RPC
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/59777.

In earlier iterations of our implementation of Git object deduplication
we thought we would be making extensive use of Git remotes in pool
repositories in the future, and that we should manage these remotes
carefully from the start. We now expect we only care about one remote,
namely the source project. The other remotes are there only for forensic
purposes.

Before this MR we tried to also remove pool remotes when member projects
got deleted, with the UnlinkRepositoryFromObjectPool RPC. This is
fragile when there are race conditions (see
https://gitlab.com/gitlab-org/gitaly/issues/1568#note_153955926). We
have spent some time making this RPC less fragile in
https://gitlab.com/gitlab-org/gitaly/merge_requests/1151 but looking at
this problem again, I think we should just stop calling it.
2019-04-02 13:20:26 +00:00
Nick Thomas
013f7cd24c
Inherit from ApplicationRecord instead of ActiveRecord::Base 2019-03-28 16:18:23 +00:00
Stan Hu
e5990db539 Use Project#full_path instead of Project#path_with_namespace 2019-02-05 21:33:10 -08:00
Stan Hu
82b6e5378a Send project name with Gitaly repository requests
When hashed storage is in use, it's helpful to have the project
name associated with the request.

Closes https://gitlab.com/gitlab-org/gitaly/issues/1394
2019-02-05 21:33:09 -08:00
Zeger-Jan van de Weg
e672ed8494
Nullify pool_repository when a project leaves
Previously the nullification wasn't done, as the only caller would later
destroy the model all together. In
https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9070 this change
was made. This commit is basically a backport before that MR is merged.
2019-01-11 11:50:44 +01:00
Zeger-Jan van de Weg
752e9c18a1
Leave object pools when destroying projects
This action doesn't lean on reduplication, so a short call can me made
to the Gitaly server to have the object pool remove its remote to the
project pending deletion.
https://gitlab.com/gitlab-org/gitaly/blob/f6cd55357/internal/git/objectpool/link.go#L58

When an object pool doesn't have members, this would invalidate the need
for a pool. So when a project leaves the pool, the pool will be
destroyed on the background.

Fixes: https://gitlab.com/gitlab-org/gitaly/issues/1415
2018-12-19 13:21:56 +01:00
Zeger-Jan van de Weg
89a407dc3b
Restore Object Pools when restoring an object pool
Pool repositories are persisted in the database, and when the DB is
restored, the data need to be restored on disk. This is done by
resetting the state machine and rescheduling the object pool creation.

This is not an exact replica of the state like at the time of the
creation of the backup. However, the data is consistent again.

Dumping isn't required as internally GitLab uses git bundles which
bundle all refs and include all objects in the bundle that they require,
reduplicating as more repositories get backed up. This does require more
data to be stored.

Fixes https://gitlab.com/gitlab-org/gitaly/issues/1355
2018-12-13 09:00:03 +01:00
Zeger-Jan van de Weg
896c0bdbfb
Allow public forks to be deduplicated
When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.

Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.

When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.

These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.

Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.

To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.

Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)

In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.
2018-12-07 19:18:37 +01:00
Toon Claes
198fdc5478 Store hashed storage paths in the database 2018-11-29 16:08:14 +00:00
Zeger-Jan van de Weg
e6297a029f Use #update! over #update to make errors visible 2018-11-28 08:29:46 +00:00
Zeger-Jan van de Weg
fff7754186
Rename the Repository table to PoolRepository
To separate the different kinds of repositories we have at GitLab this
table will be renamed to pool_repositories. A project can, for now at
least, be member of none, or one of these. The table will get additional
columns in a later merge request where more logic is implemented for the
model.

Further included is a small refactor of logic around hashing ids for the
disk_path, mainly to ensure a previous implementation is reusable.

The disk_path for the pool_repositories table no longer has a NOT NULL
constraint, but given the hashing of the ID requires the DB to assign
the record an ID, an after_create hook is used to update the value.

A related MR is:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23143, adding
tables for 'normal' repositories and wiki_repositories.
2018-11-27 13:41:46 +01:00
Nick Thomas
270155d655
Start tracking pool repositories 2018-11-05 01:40:29 +00:00