moby--moby/reports/2017-06-26.md

Development Report for June 26, 2017

Moby Summit

The Moby Summit held in San Francisco was very active and well attended (blog / linuxkit table notes #2090 #2033 [@mgoelzer] [@justincormack]).

Container Engine

Thanks to @fabiokung there is no container locks anymore on docker ps #31273

BuildKit

Repo Proposal

New development repo is open at https://github.com/moby/buildkit

The readme file provides examples how to get started. You can see an example of building BuildKit with BuildKit.

There are lots of new issues opened as well to track the missing functionality. You are welcomed to help on any of them or discuss the design there.

Last week most of the work was done on improving the llb client library for more complicated use cases and providing traces and interactive progress of executed build jobs.

The llb client package is a go library that helps you to generate the build definition graph. It uses chained methods to make it easy to describe what steps need to be running. Mounts can be added to the execution steps for defining multiple inputs or outputs. To prepare the graph, you just have to call Marshal() on a leaf node that will generate the protobuf definition for everything required to build that node.

Typed Dockerfile parsing

PR

This PR that enables parsing Dockerfiles into typed structures so they can be preprocessed to eliminate unnecessary build stages and reused with different kinds of dispatchers(eg. BuildKit).

The PR had some review and updates in last week. Should be ready to code review soon.

Merged: Long running session & incremental file sending

PR

Incremental context sending PR was merged and is expected to land in v17.07.

This feature experimental feature lets you skip sending the build context to the daemon on repeated builder invocations during development. Currently, this feature requires a CLI flag --stream=true. If this flag is used, one first builder invocation full build context is sent to the daemon. On a second attempt, only the changed files are transferred.

Previous build context is saved in the build cache, and you can see how much space it takes form docker system df. Build cache will be automatically garbage collected and can also be manually cleared with docker prune.

Quality: Dependency interface switch

Move file copying from the daemon to the builder PR was merged.

Proposals for new Dockerfile features that need design feedback:

Add IMPORT/EXPORT commands to Dockerfile

Add DOCKEROS/DOCKERARCH default ARG to Dockerfile

Add support for RUN --mount

DAG image builder

Option to export the hash of the build context (new)

Allow --cache-from=* (new)

Provide advanced .dockeringore use-cases 2

If you are interested in implementing any of them, leave a comment on the specific issues.

Other builder PRs merged last week

Warn/deprecate continuing on empty lines in Dockerfile

Fix behavior of absolute paths in .dockerignore

fix copy —from conflict with force pull

Builder features currently in code-review:

Fix handling of remote "git@" notation

builder: Emit a BuildResult after squashing.

Fix shallow git clone in docker-build

Backlog

Build secrets has not got much traction. If you want this feature to become a reality, please make yourself heard.

LinuxKit

• Kernel GPG verification: The kernel compilation containers now verify the GPG and SHA256 checksums before building the binaries. (#2062 #2083 [@mscribe] [@justincormack] [@rn] [@riyazdf]). The base Alpine build image now includes gnupg to support this feature (#2091 [@riyazdf] [@rn]).

• Security SIG on Landlock: The third Moby Security SIG focussed on the Landlock security module that provides unprivileged fine-grained sandboxing to applications. There are videos and forum links (#2087 #2089 #2073 [@riyazdf]).

• Networking drivers now modules: The kernels have been updated to 4.11.6/4.9.33/4.4.73, and many drivers are now loaded as modules to speed up boot-time (#2095 #2061 [@rn] [@justincormack] [@tych0])

• Whaley important update: The ASCII logo was updated and we fondly wave goodbye to the waves. (#2084 [@thaJeztah] [@rn])

• Containerised getty and sshd: The login services now run in their own mount namespace, which was confusing people since they were expecting it to be on the host filesystem. This is now being addressed via a reminder in the motd upon login (#2078 #2097 [@deitch] [@ijc] [@justincormack] [@riyazdf] [@rn])

• Hardened user copying: The RFC on ensuring that we use a hardened kernel/userspace copying system was closed, as it is enabled by default on all our modern kernels and a regression test is included by default (#2086 [@fntlnz] [@riyazdf]).

• Vultr provider: There is an ongoing effort to add a metadata provider for Vultr (#2101 [@furious-luke] [@justincormack]).

Packages and Projects

• Simplified Makefiles for packages (#2080 [@justincormack] [@rn])
• The MirageOS SDK is integrating many upstream changes from dependent libraries, for the DHCP client (#2070 #2072 [@samoht] [@talex5] [@avsm]).

Documentation and Tests

• A comprehensive test suite for containerd is now integrated into LinuxKit tests (#2062 [@AkihiroSuda] [@justincormack] [@rn])
• Fix documentation links (#2074 [@ndauten] [@justincormack])
• Update RTF version (#2077 [@justincormack])
• tests: add build test for Docker for Mac blueprint (#2093 [@riyazdf] [@MagnusS])
• Disable Qemu EFI ISO test for now (#2100 [@justincormack])
• The CI whitelists and ACLs were updated (linuxkit-ci#11 linuxkit-ci#15 linuxkit/linuxkit-ci#10 [@rn] [@justincormack])
• Fix spelling errors (#2079 [@ndauten])
• Fix typo in dev report (#2094 [@justincormack])
• Fix dead Link to VMWare File (#2082 [@davefreitag])