Commit graph

23 commits

Author SHA1 Message Date
Stan Hu
25818bd7ae Disable method replacement in avatar loading
We've seen a significant performance penalty when using
`BatchLoader#__replace_with!`. This defines methods on the batch loader
that proxy to the 'real' object using send. The alternative is
`method_missing`, which is slower.  However, we've noticed that
`method_missing` can be faster if:

1. The objects being loaded have a large interface.
2. We don't call too many methods on the loaded object.

Avatar uploads meet both criteria above, so let's use the newly-released
feature in https://github.com/exAspArk/batch-loader/pull/45.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/60903
2019-04-29 21:29:25 -07:00
Bastian Blank
b82ccf2cbc Don't try to set any ACL on uploaded objects
Set ACL of uploaded objects to not-public (fog-aws) or no ACL at all
(fog-google).  Value is ignored by other supported backends (fog-aliyun,
fog-openstack, fog-rackspace)

This allows uploads to GCS with ACL support disabled.
2019-03-30 14:55:33 +01:00
Michael Kozono
d2d85c31fe Avoid extra storage bucket perm and query
Specifically, the `ListAllMyBuckets` permission.

This works if you know the directory exists.

See more:

* https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23981
* https://stackoverflow.com/a/12288581/1992201
2018-12-21 10:34:05 -08:00
Lin Jen-Shin
6037c17cd4 Bring this test from EE 2018-06-21 21:39:28 +08:00
Sean McGivern
a66af9b121 Merge branch '47513-upload-migration-lease-key-is-incorrect-for-non-mounted-uploaders' into 'master'
Resolve "Upload migration lease key is incorrect for non-mounted uploaders"

Closes #47513

See merge request gitlab-org/gitlab-ce!19600
2018-06-13 09:45:34 +00:00
Sean McGivern
8d9e0c0b8c Merge branch '47408-migrateuploadsworker-is-doing-n-1-queries-on-migration' into 'master'
Resolve "`MigrateUploadsWorker` is doing N+1 queries on migration"

Closes #47408

See merge request gitlab-org/gitlab-ce!19547
2018-06-13 08:30:35 +00:00
Jan Provaznik
c3f499e7c8 Use upload ID instead of model ID in lease key
For FileUploaders it's possible that a model has many uploads
and if lease key is created only from model id, it causes that
the model's uploads can not be migrated in parallel because the
exclusive lease key would be same for all uploads of the model.
2018-06-12 09:34:07 +02:00
Micaël Bergeron
50872bcc24
fix the failing spec 2018-06-07 12:06:00 -04:00
Robert Speicher
6d165c740c Make all uses of fixture_file_upload use relative paths 2018-06-07 09:54:41 -05:00
Rémy Coutable
0a5f3aa378 Merge branch 'fix-fog-mocking' into 'master'
Fix Fog mocking

Closes #47300

See merge request gitlab-org/gitlab-ce!19425
2018-06-05 15:26:14 +00:00
Kamil Trzciński
53d1c87c86 Fix Fog mocking 2018-06-05 16:48:42 +02:00
Sean McGivern
6ecf819f73 Fix an N+1 in avatar URLs
This is tricky: the query was being run in
`ObjectStorage::Extension::RecordsUploads#retrieve_from_store!`, but we can't
just add batch loading there, because the `#upload=` method there would use the
result immediately, making the batch only have one item.

Instead, we can pre-emptively add an item to the batch whenever an avatarable
object is initialized, and then reuse that batch item in
`#retrieve_from_store!`. However, this also has problems:

1. There is a lot of logic in `Avatarable#retrieve_upload_from_batch`.
2. Some of that logic constructs a 'fake' model for the batch key. This should
   be fine, because of ActiveRecord's override of `#==`, but it relies on that
   staying the same.
2018-06-05 14:57:19 +01:00
Kamil Trzciński
b8370c9f55 Support presigned multipart uploads 2018-06-04 13:04:29 +02:00
Alessio Caiazza
a23ef89e57
Fix missing timeout value in object storage pre-authorization call 2018-05-29 19:01:05 +02:00
Shinya Maeda
cb2802dbbc Fix spec 2018-05-28 10:44:26 +09:00
Kamil Trzciński
a28f25b565 Fix direct_upload when records with null file_store are used
Old records have a null value of file_store column.
This causes the problems with current direct_upload implementation,
as this makes it to choose Store::REMOTE instead of Store::LOCAL.

This change moves the store save when change saving the object.
2018-04-19 12:09:51 +02:00
Kamil Trzciński
678620cce6 Add direct_upload setting for artifacts 2018-04-05 15:01:14 +02:00
Kamil Trzciński
9dde7df247 Allow to store uploads by default on Object Storage
Introduce `direct_upload` option for `uploads` which is gonna set a default storage to Object Storage and use Unicorn to save data
2018-04-04 23:13:45 +02:00
Alessio Caiazza
ae27a47b6e
Add Gitlab::ExclusiveLease to ObjectStorage#use_file 2018-03-29 10:01:23 +02:00
Alessio Caiazza
04c5e637f8
Port LFS direct_upload from EE 2018-03-27 10:32:48 +02:00
Micaël Bergeron
508938873c remove the license check 2018-03-01 10:40:40 -05:00
Kamil Trzciński
a22f6fa6e5 Merge branch 'fix/sm/atomic-migration' into 'master'
Fix migrate! method (Minimal fix with ExclusiveLock to prevent race conditions)

Closes #4928 and #4980

See merge request gitlab-org/gitlab-ee!4624
2018-02-28 21:28:26 +01:00
Sean McGivern
a7dae52e9d Merge branch '4163-move-uploads-to-object-storage' into 'master'
Move uploads to object storage

Closes #4163

See merge request gitlab-org/gitlab-ee!3867
2018-02-28 20:58:15 +01:00