forks
/
peertube
1
0
Fork 0
Code Releases Activity
peertube/server/initializers/migrations
History
Jelle Besseling 0305db28c9
Add support for saving video files to object storage (#4290) 
* Add support for saving video files to object storage

* Add support for custom url generation on s3 stored files

Uses two config keys to support url generation that doesn't directly go
to (compatible s3). Can be used to generate urls to any cache server or
CDN.

* Upload files to s3 concurrently and delete originals afterwards

* Only publish after move to object storage is complete

* Use base url instead of url template

* Fix mistyped config field

* Add rudenmentary way to download before transcode

* Implement Chocobozzz suggestions

https://github.com/Chocobozzz/PeerTube/pull/4290#issuecomment-891670478

The remarks in question:
    Try to use objectStorage prefix instead of s3 prefix for your function/variables/config names
    Prefer to use a tree for the config: s3.streaming_playlists_bucket -> object_storage.streaming_playlists.bucket
    Use uppercase for config: S3.STREAMING_PLAYLISTS_BUCKETINFO.bucket -> OBJECT_STORAGE.STREAMING_PLAYLISTS.BUCKET (maybe BUCKET_NAME instead of BUCKET)
    I suggest to rename moveJobsRunning to pendingMovingJobs (or better, create a dedicated videoJobInfo table with a pendingMove & videoId columns so we could also use this table to track pending transcoding jobs)
    https://github.com/Chocobozzz/PeerTube/pull/4290/files#diff-3e26d41ca4bda1de8e1747af70ca2af642abcc1e9e0bfb94239ff2165acfbde5R19 uses a string instead of an integer
    I think we should store the origin object storage URL in fileUrl, without base_url injection. Instead, inject the base_url at "runtime" so admins can easily change this configuration without running a script to update DB URLs

* Import correct function

* Support multipart upload

* Remove import of node 15.0 module stream/promises

* Extend maximum upload job length

Using the same value as for redundancy downloading seems logical

* Use dynamic part size for really large uploads

Also adds very small part size for local testing

* Fix decreasePendingMove query

* Resolve various PR comments

* Move to object storage after optimize

* Make upload size configurable and increase default

* Prune webtorrent files that are stored in object storage

* Move files after transcoding jobs

* Fix federation

* Add video path manager

* Support move to external storage job in client

* Fix live object storage tests

Co-authored-by: Chocobozzz <me@florianbigard.com>
 		2021-08-17 08:26:20 +02:00
..
0005-email-pod.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0010-email-user.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0015-video-views.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0020-video-likes.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0025-video-dislikes.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0030-video-category.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0035-video-licence.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0040-video-nsfw.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0045-user-display-nsfw.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0050-video-language.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0055-video-uuid.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0060-video-file.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0065-video-file-size.ts Add support for saving video files to object storage (#4290) 2021-08-17 08:26:20 +02:00
0070-user-video-quota.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0075-video-resolutions.ts Improve target bitrate calculation 2021-08-06 14:13:26 +02:00
0080-video-channels.ts Support short uuid for GET video/playlist 2021-06-29 14:56:35 +02:00
0085-user-role.ts Move to eslint 2020-02-03 08:31:02 +01:00
0090-videos-description.ts Move to eslint 2020-02-03 08:31:02 +01:00
0095-videos-privacy.ts Move to eslint 2020-02-03 08:31:02 +01:00
0100-activitypub.ts Fix incorrect IDs in AP federation 2020-11-20 11:21:08 +01:00
0105-server-mail.ts Move to eslint 2020-02-03 08:31:02 +01:00
0110-server-key.ts Move to eslint 2020-02-03 08:31:02 +01:00
0115-account-avatar.ts Move to eslint 2020-02-03 08:31:02 +01:00
0120-video-null.ts Move to eslint 2020-02-03 08:31:02 +01:00
0125-table-lowercase.ts Move to eslint 2020-02-03 08:31:02 +01:00
0130-user-autoplay-video.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0135-video-channel-actor.ts Fix build with typescript 3.8 2020-02-28 16:51:25 +01:00
0140-actor-url.ts Move to eslint 2020-02-03 08:31:02 +01:00
0145-delete-author.ts Move to eslint 2020-02-03 08:31:02 +01:00
0150-avatar-cascade.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0155-video-comments-enabled.ts Move to eslint 2020-02-03 08:31:02 +01:00
0160-account-route.ts Move to eslint 2020-02-03 08:31:02 +01:00
0165-video-route.ts Move to eslint 2020-02-03 08:31:02 +01:00
0170-actor-follow-score.ts Move to eslint 2020-02-03 08:31:02 +01:00
0175-actor-follow-counts.ts Move to eslint 2020-02-03 08:31:02 +01:00
0180-job-table-delete.ts Move to eslint 2020-02-03 08:31:02 +01:00
0185-video-share-url.ts Move to eslint 2020-02-03 08:31:02 +01:00
0190-video-comment-unique-url.ts Move to eslint 2020-02-03 08:31:02 +01:00
0195-support.ts Move to eslint 2020-02-03 08:31:02 +01:00
0200-video-published-at.ts Move to eslint 2020-02-03 08:31:02 +01:00
0205-user-nsfw-policy.ts Move to eslint 2020-02-03 08:31:02 +01:00
0210-video-language.ts Move to eslint 2020-02-03 08:31:02 +01:00
0215-video-support-length.ts Move to eslint 2020-02-03 08:31:02 +01:00
0220-video-state.ts
…
0225-video-fps.ts
…
0235-delete-some-video-indexes.ts Move config in its own file 2019-04-11 13:45:39 +02:00
0240-drop-old-indexes.ts Move config in its own file 2019-04-11 13:45:39 +02:00
0245-user-blocked.ts Feature/description support fields length 1000 (#1267) 2018-10-19 08:54:01 +02:00
0250-video-abuse-state.ts Use 3 tables to represent abuses 2020-07-10 14:02:41 +02:00
0255-video-blacklist-reason.ts Move to eslint 2020-02-03 08:31:02 +01:00
0260-upload-quota-daily.ts Feature/description support fields length 1000 (#1267) 2018-10-19 08:54:01 +02:00
0265-user-email-verified.ts
…
0270-server-redundancy.ts Basic video redundancy implementation 2018-09-13 14:05:49 +02:00
0275-video-file-unique.ts Avoid old issue regarding duplicated hosts in db 2018-10-08 11:58:18 +02:00
0280-webtorrent-policy-user.ts move to boolean switch 2018-10-13 11:53:48 +02:00
0285-description-support.ts Move to eslint 2020-02-03 08:31:02 +01:00
0290-account-video-rate-url.ts Move to eslint 2020-02-03 08:31:02 +01:00
0295-video-file-extname.ts Move to eslint 2020-02-03 08:31:02 +01:00
0300-user-videos-history-enabled.ts Move to eslint 2020-02-03 08:31:02 +01:00
0305-fix-unfederated-videos.ts Move to eslint 2020-02-03 08:31:02 +01:00
0310-drop-unused-video-indexes.ts Move to eslint 2020-02-03 08:31:02 +01:00
0315-user-notifications.ts Move to eslint 2020-02-03 08:31:02 +01:00
0320-blacklist-unfederate.ts Move to eslint 2020-02-03 08:31:02 +01:00
0325-video-abuse-fields.ts Move to eslint 2020-02-03 08:31:02 +01:00
0330-video-streaming-playlist.ts Move to eslint 2020-02-03 08:31:02 +01:00
0335-video-downloading-enabled.ts Move to eslint 2020-02-03 08:31:02 +01:00
0340-add-originally-published-at.ts Move to eslint 2020-02-03 08:31:02 +01:00
0345-video-playlists.ts Support short uuid for GET video/playlist 2021-06-29 14:56:35 +02:00
0350-video-blacklist-type.ts Move to eslint 2020-02-03 08:31:02 +01:00
0355-p2p-peer-version.ts Move to eslint 2020-02-03 08:31:02 +01:00
0360-notification-instance-follower.ts Move to eslint 2020-02-03 08:31:02 +01:00
0365-user-admin-flags.ts Move to eslint 2020-02-03 08:31:02 +01:00
0370-thumbnail.ts Move to eslint 2020-02-03 08:31:02 +01:00
0375-account-description.ts Move to eslint 2020-02-03 08:31:02 +01:00
0380-cleanup-timestamps.ts Move to eslint 2020-02-03 08:31:02 +01:00
0385-remove-actor-uuid.ts Move to eslint 2020-02-03 08:31:02 +01:00
0390-user-pending-email.ts Move to eslint 2020-02-03 08:31:02 +01:00
0395-user-video-languages.ts Move to eslint 2020-02-03 08:31:02 +01:00
0400-user-theme.ts Move to eslint 2020-02-03 08:31:02 +01:00
0405-plugin.ts Move to eslint 2020-02-03 08:31:02 +01:00
0410-video-playlist-element.ts Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
0415-thumbnail-auto-generated.ts Move to eslint 2020-02-03 08:31:02 +01:00
0420-avatar-lazy.ts Move to eslint 2020-02-03 08:31:02 +01:00
0425-nullable-actor-fields.ts Move to eslint 2020-02-03 08:31:02 +01:00
0430-auto-follow-notification-setting.ts Move to eslint 2020-02-03 08:31:02 +01:00
0435-user-modals.ts Move to eslint 2020-02-03 08:31:02 +01:00
0440-user-auto-play-next-video.ts Move to eslint 2020-02-03 08:31:02 +01:00
0445-shared-inbox-optional.ts Move to eslint 2020-02-03 08:31:02 +01:00
0450-streaming-playlist-files.ts Move to eslint 2020-02-03 08:31:02 +01:00
0455-soft-delete-video-comments.ts Move to eslint 2020-02-03 08:31:02 +01:00
0460-user-playlist-autoplay.ts Move to eslint 2020-02-03 08:31:02 +01:00
0465-thumbnail-file-url-length.ts Move to eslint 2020-02-03 08:31:02 +01:00
0470-cleanup-indexes.ts Add migrations 2020-07-10 14:02:41 +02:00
0475-redundancy-expires-on.ts Move to eslint 2020-02-03 08:31:02 +01:00
0480-caption-file-url.ts Move to eslint 2020-02-03 08:31:02 +01:00
0490-abuse-video.ts Factorize rest-table and fix/simplify SQL 2020-05-01 16:41:02 +02:00
0495-plugin-auth.ts Fix plugin-auth migration 2020-05-05 13:52:10 +02:00
0500-playlist-description-length.ts Fix playlist description length 2020-05-05 16:34:50 +02:00
0505-user-last-login-date.ts Add last login date to users 2020-05-07 10:39:09 +02:00
0510-video-file-metadata.ts Fix broken migration introduced in 2.2.0-rc.1 2020-05-25 08:59:59 +02:00
0515-video-abuse-reason-timestamps.ts predefined report reasons & improved reporter UI (#2842) 2020-06-22 13:00:39 +02:00
0520-abuses-split.ts Relax migrations 2020-07-11 08:58:06 +02:00
0525-abuse-messages.ts Add migrations for abuse messages 2020-07-31 11:35:19 +02:00
0530-playlist-multiple-video.ts Add server migration for playlists 2020-08-19 11:30:21 +02:00
0535-video-live.ts Check live duration and size 2020-11-09 15:33:04 +01:00
0540-video-file-infohash.ts Live streaming implementation first step 2020-11-09 15:33:04 +01:00
0545-video-live-save-replay.ts Fix live migrations 2020-11-10 10:45:20 +01:00
0550-actor-follow-cleanup.ts Cleanup follows of orphean actors 2020-11-10 16:29:35 +01:00
0555-actor-follow-url.ts Add actor follow url migration 2020-11-20 12:09:14 +01:00
0560-user-feed-token.ts Support short uuid for GET video/playlist 2021-06-29 14:56:35 +02:00
0565-actor-follow-local-url.ts Add local actor follow url migration 2020-12-01 15:04:38 +01:00
0570-permanent-live.ts Add permanent live support 2020-12-03 15:21:16 +01:00
0575-duplicate-thumbnail.ts Generate a name for thumbnails 2021-02-16 10:36:44 +01:00
0580-caption-filename.ts Generate a name for caption files 2021-02-16 10:36:44 +01:00
0585-video-file-names.ts Add video files migration 2021-02-18 13:38:09 +01:00
0590-trackers.ts Don't guess remote tracker URL 2021-02-18 13:38:09 +01:00
0595-remote-url.ts Don't guess remote tracker URL 2021-02-18 13:38:09 +01:00
0600-duplicate-video-files.ts Fix latest migrations 2021-02-19 08:41:32 +01:00
0605-actor-missing-keys.ts Fix broken local actors 2021-02-26 14:22:25 +01:00
0610-views-index copy.ts Add new plugin/peertube version notifs 2021-03-24 18:18:41 +01:00
0612-captions-unique.ts Remove duplicate captions 2021-04-08 14:45:40 +02:00
0615-latest-versions-notification-settings.ts Add new plugin/peertube version notifs 2021-03-24 18:18:41 +01:00
0620-latest-versions-application.ts Add new plugin/peertube version notifs 2021-03-24 18:18:41 +01:00
0625-latest-versions-notification.ts Add new plugin/peertube version notifs 2021-03-24 18:18:41 +01:00
0630-banner.ts Add banner migrations 2021-04-08 10:07:53 +02:00
0635-actor-image-size.ts Add size info in db for actor images 2021-04-08 13:38:04 +02:00
0640-unique-keys.ts Add migration to remove duplicated keys 2021-04-26 14:27:42 +02:00
0645-actor-remote-creation-date.ts Fix remote actor creation date 2021-05-07 09:00:09 +02:00
0650-actor-custom-pages.ts Instance homepage support (#4007) 2021-05-27 15:59:55 +02:00
0655-streaming-playlist-filenames.ts Use random names for VOD HLS playlists 2021-07-26 11:29:31 +02:00
0660-object-storage.ts Add support for saving video files to object storage (#4290) 2021-08-17 08:26:20 +02:00