4 KiB
Development Report for July 10, 2017
BuildKit
Many new features have been added since the last report.
The build definition solver was updated to detect the identical parts of the graph sent by different clients and synchronize their processing. This is important when multiple targets of the same project are built at the same time and removes any duplication of work.
Running build jobs now has support for graceful canceling and clear error reporting in case some build steps fail or are canceled. Bugs that may have left state dir in an inconsistent state of server shutdown were fixed.
buildctl du command now shows all the information about allocated and in-use snapshots. It also shows the total space used and total reclaimable space. All snapshots are now persistent, and state is not lost with server restarts.
New metadata package was implemented that other packages can use to add persistent and searchable metadata to individual snapshots. First users of that feature are the content blobs mapping on pull, size cache for du and instruction cache. There is also a new debug command buildctl debug dump-metadata to inspect what data is being stored.
The first version of instruction cache was implemented. This caching scheme has many benefits compared to the current docker build caching as it doesn't require all data to be locally available to determine the cache match. The interface for the cache implementation is much simpler and could be implemented remotely as it only needs to store the cache keys and doesn't need to understand or compare their values. Content-based caching will be implemented on top of this work later.
Separate source implementation for git repositories is currently in review. Using this source for accessing source code in git repositories has many performance and caching advantages. All the build jobs using the same git remote will use a shared local repository where updates will be pulled. All the nodes based on a git source will be cached using the commit ID of the current checkout.
Next areas to be worked on will be implementing first exporters for getting access to the build artifacts and porting over the client session/incremental-send feature from 17.07-ce.
Typed Dockerfile parsing
The PR is in code review and waiting for feedback. Hopefully ready to be merged this week.
Quality: Dependency interface switch
No updates for this week. Metadata commands need to be updated but it is probably easier to do it after https://github.com/moby/moby/pull/33492 has been merged.
Proposals for new Dockerfile features that need design feedback:
Add IMPORT/EXPORT commands to Dockerfile
Add DOCKEROS/DOCKERARCH default ARG to Dockerfile
Option to export the hash of the build context (new)
Allow --cache-from=* (new)
Provide advanced .dockeringore use-cases 2
New: RFC: Distributed BuildKit
If you are interested in implementing any of them, leave a comment on the specific issues.
Other builder PRs merged last week
build: fix add from remote url
Builder features currently in code-review:
Fix shallow git clone in docker-build
Backlog
Build secrets has not got much traction. If you want this feature to become a reality, please make yourself heard.