Tracks source repository information for each blob in the blobsum
service, which is then used to attempt to mount blobs from another
repository when pushing instead of having to re-push blobs to the same
registry.
Signed-off-by: Brian Bland <brian.bland@docker.com>
A manifest list refers to platform-specific manifests. This allows
for images that target more than one architecture to share the same tag.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This is a followup to #18839. That PR relaxed the fallback logic so that
if a manifest doesn't exist on v2, or the user is unauthorized to access
it, we try again with the v1 protocol. A similar special case is needed
for "pull all tags" (docker pull -a). If the v2 registry doesn't
recognize the repository, or doesn't allow the user to access it, we
should fall back to v1 and try to pull all tags from the v1 registry.
Conversely, if the v2 registry does allow us to list the tags, there
should be no fallback, even if there are errors pulling those tags.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
PR #18590 caused compatibility issues with registries such as gcr.io
which support both the v1 and v2 protocols, but do not provide the same
set of images over both protocols. After #18590, pulls from these
registries would never use the v1 protocol, because of the
Docker-Distribution-Api-Version header indicating that v2 was supported.
Fix the problem by making an exception for the case where a manifest is
not found. This should allow fallback to v1 in case that image is
exposed over the v1 protocol but not the v2 protocol.
This avoids the overly aggressive fallback behavior before #18590 which
would allow protocol fallback after almost any error, but restores
interoperability with mixed v1/v2 registry setups.
Fixes#18832
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
If we detect a Docker-Distribution-Api-Version header indicating that
the registry speaks the V2 protocol, no fallback to V1 should take
place.
The same applies if a V2 registry operation succeeds while attempting a
push or pull.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
This commit adds a transfer manager which deduplicates and schedules
transfers, and also an upload manager and download manager that build on
top of the transfer manager to provide high-level interfaces for uploads
and downloads. The push and pull code is modified to use these building
blocks.
Some benefits of the changes:
- Simplification of push/pull code
- Pushes can upload layers concurrently
- Failed downloads and uploads are retried after backoff delays
- Cancellation is supported, but individual transfers will only be
cancelled if all pushes or pulls using them are cancelled.
- The distribution code is decoupled from Docker Engine packages and API
conventions (i.e. streamformatter), which will make it easier to split
out.
This commit also includes unit tests for the new distribution/xfer
package. The tests cover 87.8% of the statements in the package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
We were calling Stat for each layer to get the size so we could indicate
progress, but https://github.com/docker/distribution/pull/1226 made it
possible to get the length from the GET request that Open initiates.
Saving one round-trip per layer should make pull operations slightly
faster and more robust.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Brian Bland