kotovalexarian-likes-github/moby--moby
1
0
Fork 0
mirror of https://github.com/moby/moby.git synced 2022-11-09 12:21:53 -05:00
Code Releases Activity
moby--moby/daemon
History
Kir Kolyshkin ae431b10a9 aufs: retry auplink flush 
Running a bundled aufs benchmark sometimes results in this warning:

> WARN[0001] Couldn't run auplink before unmount /tmp/aufs-tests/aufs/mnt/XXXXX  error="exit status 22" storage-driver=aufs

If we take a look at what aulink utility produces on stderr, we'll see:

> auplink:proc_mnt.c:96: /tmp/aufs-tests/aufs/mnt/XXXXX: Invalid argument

and auplink exits with exit code of 22 (EINVAL).

Looking into auplink source code, what happens is it tries to find a
record in /proc/self/mounts corresponding to the mount point (by using
setmntent()/getmntent_r() glibc functions), and it fails.

Some manual testing, as well as runtime testing with lots of printf
added on mount/unmount, as well as calls to check the superblock fs
magic on mount point (as in graphdriver.Mounted(graphdriver.FsMagicAufs, target)
confirmed that this record is in fact there, but sometimes auplink
can't find it. I was also able to reproduce the same error (inability
to find a mount in /proc/self/mounts that should definitely be there)
using a small C program, mocking what `auplink` does:

```c
 #include <stdio.h>
 #include <err.h>
 #include <mntent.h>
 #include <string.h>
 #include <stdlib.h>

int main(int argc, char **argv)
{
	FILE *fp;
	struct mntent m, *p;
	char a[4096];
	char buf[4096 + 1024];
	int found =0, lines = 0;

	if (argc != 2) {
		fprintf(stderr, "Usage: %s <mountpoint>\n", argv[0]);
		exit(1);
	}

	fp = setmntent("/proc/self/mounts", "r");
	if (!fp) {
		err(1, "setmntent");
	}
	setvbuf(fp, a, _IOLBF, sizeof(a));
	while ((p = getmntent_r(fp, &m, buf, sizeof(buf)))) {
		lines++;
		if (!strcmp(p->mnt_dir, argv[1])) {
			found++;
		}
	}
	printf("found %d entries for %s (%d lines seen)\n", found, argv[1], lines);
	return !found;
}
```

I have also wrote a few other C proggies -- one that reads
/proc/self/mounts directly, one that reads /proc/self/mountinfo instead.
They are also prone to the same occasional error.

It is not perfectly clear why this happens, but so far my best theory
is when a lot of mounts/unmounts happen in parallel with reading
contents of /proc/self/mounts, sometimes the kernel fails to provide
continuity (i.e. it skips some part of file or mixes it up in some
other way). In other words, this is a kernel bug (which is probably
hard to fix unless some other interface to get a mount entry is added).

Now, there is no real fix, and a workaround I was able to come up
with is to retry when we got EINVAL. It usually works on the second
attempt, although I've once seen it took two attempts to go through.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2019-05-21 10:58:59 -07:00
..
cluster Merge pull request #39190 from ollypom/swarmnanocpu 2019-05-20 10:14:07 -07:00
config Make cgroup namespaces configurable 2019-05-07 10:22:16 -07:00
discovery Update tests to use gotest.tools 👼 2018-06-13 09:04:30 +02:00
events Add canonical import comment 2018-02-05 16:51:57 -05:00
exec Fix race condition between exec start and resize 2018-06-08 11:07:48 +08:00
graphdriver aufs: retry auplink flush 2019-05-21 10:58:59 -07:00
images Merge pull request #37534 from thaJeztah/fix-distribution-500 2019-03-13 08:29:16 +01:00
initlayer Add ADD/COPY --chown flag support to Windows 2018-08-13 21:59:11 -07:00
links fix typo 2018-09-08 08:13:30 +08:00
listeners allow running dockerd in an unprivileged user namespace (rootless mode) 2019-02-04 00:24:27 +09:00
logger Merge pull request #38586 from robin-thoni/labels-regex 2019-04-29 12:25:21 -07:00
names Add canonical import comment 2018-02-05 16:51:57 -05:00
network Network: add support for 'dangling' filter 2019-02-27 15:08:44 -05:00
stats Merge pull request #38510 from ZYecho/tune-code 2019-03-21 13:56:02 -07:00
testdata Remove libtrust dep from api 2017-09-06 12:05:19 -04:00
apparmor_default.go Add canonical import comment 2018-02-05 16:51:57 -05:00
apparmor_default_unsupported.go Add canonical import comment 2018-02-05 16:51:57 -05:00
archive.go Add canonical import comment 2018-02-05 16:51:57 -05:00
archive_tarcopyoptions.go Add ADD/COPY --chown flag support to Windows 2018-08-13 21:59:11 -07:00
archive_tarcopyoptions_unix.go Add ADD/COPY --chown flag support to Windows 2018-08-13 21:59:11 -07:00
archive_tarcopyoptions_windows.go Add canonical import comment 2018-02-05 16:51:57 -05:00
archive_unix.go Move mount parsing to separate package. 2018-04-19 06:35:54 -04:00
archive_windows.go Add canonical import comment 2018-02-05 16:51:57 -05:00
attach.go daemon.ContainerLogs(): fix resource leak on follow 2018-09-06 11:47:42 -07:00
auth.go Switch from x/net/context -> context 2018-04-23 13:52:44 -07:00
changes.go c.RWLayer: check for nil before use 2018-02-09 11:24:09 -08:00
checkpoint.go Merge pull request #38405 from rst0git/enable-cr-tty 2019-03-21 14:12:49 -07:00
cluster.go Move network conversions out of API router 2018-06-27 17:11:29 -07:00
commit.go Windows: (WCOW) Generate OCI spec that remote runtime can escape 2019-03-12 18:41:55 -07:00
configs.go Merge configs/secrets in unix implementation 2018-02-16 11:25:14 -05:00
configs_linux.go Add canonical import comment 2018-02-05 16:51:57 -05:00
configs_unsupported.go Add canonical import comment 2018-02-05 16:51:57 -05:00
configs_windows.go Add canonical import comment 2018-02-05 16:51:57 -05:00
container.go Capabilities refactor 2019-01-22 21:50:41 +02:00
container_linux.go Add canonical import comment 2018-02-05 16:51:57 -05:00
container_operations.go Merge pull request #38853 from cyphar/integration-cli-ensureImage 2019-03-27 07:32:23 +01:00
container_operations_unix.go Merge pull request #38579 from thaJeztah/fix_net_host_systemd_resolved 2019-03-21 15:34:48 -07:00
container_operations_windows.go Move EnableServiceDiscoveryOnDefaultNetwork to container-operations 2019-03-20 18:45:20 +01:00
container_unix_test.go Update tests to use gotest.tools 👼 2018-06-13 09:04:30 +02:00
container_windows.go Add canonical import comment 2018-02-05 16:51:57 -05:00
create.go Merge pull request #38918 from thaJeztah/bump_selinux 2019-03-28 17:27:03 -07:00
create_test.go Update tests to use gotest.tools 👼 2018-06-13 09:04:30 +02:00
create_unix.go Add ADD/COPY --chown flag support to Windows 2018-08-13 21:59:11 -07:00
create_windows.go Extract volume interaction to a volumes service 2018-05-25 14:21:07 -04:00
daemon.go Make cgroup namespaces configurable 2019-05-07 10:22:16 -07:00
daemon_linux.go Fixes for resolv.conf 2018-07-26 11:17:56 -07:00
daemon_linux_test.go Export all spec generation opts 2019-04-10 15:38:36 -04:00
daemon_test.go Add ADD/COPY --chown flag support to Windows 2018-08-13 21:59:11 -07:00
daemon_unix.go bugfix: fetch the right device number which great than 255 2019-05-16 15:32:59 +08:00
daemon_unix_test.go bugfix: fetch the right device number which great than 255 2019-05-16 15:32:59 +08:00
daemon_unsupported.go Fixes for resolv.conf 2018-07-26 11:17:56 -07:00
daemon_windows.go Windows: Experimental: Allow containerd for runtime 2019-03-12 18:41:55 -07:00
daemon_windows_test.go Add canonical import comment 2018-02-05 16:51:57 -05:00
debugtrap_unix.go Add canonical import comment 2018-02-05 16:51:57 -05:00
debugtrap_unsupported.go Add canonical import comment 2018-02-05 16:51:57 -05:00
debugtrap_windows.go Windows:Update dumpstack event name 2019-02-15 15:26:56 -08:00
delete.go Updates daemon's remove link method to use more verbose error output. 2019-04-25 13:08:08 -04:00
delete_test.go Update tests to use gotest.tools 👼 2018-06-13 09:04:30 +02:00
dependency.go Add canonical import comment 2018-02-05 16:51:57 -05:00
devices_linux.go Add DeviceRequests to HostConfig to support NVIDIA GPUs 2019-03-18 17:19:45 +00:00
disk_usage.go Extract volume interaction to a volumes service 2018-05-25 14:21:07 -04:00
errors.go Merge pull request #38541 from Microsoft/jjh/containerd 2019-03-19 21:09:19 -07:00
events.go Image events 2018-02-21 18:26:16 -05:00
events_test.go Add canonical import comment 2018-02-05 16:51:57 -05:00
exec.go Use original process spec for execs 2019-03-21 15:41:53 -04:00
exec_linux.go Export all spec generation opts 2019-04-10 15:38:36 -04:00
exec_linux_test.go Update tests to use gotest.tools 👼 2018-06-13 09:04:30 +02:00
exec_windows.go Windows: (WCOW) Generate OCI spec that remote runtime can escape 2019-03-12 18:41:55 -07:00
export.go Add ADD/COPY --chown flag support to Windows 2018-08-13 21:59:11 -07:00
health.go Switch from x/net/context -> context 2018-04-23 13:52:44 -07:00
health_test.go Add canonical import comment 2018-02-05 16:51:57 -05:00
info.go Make cgroup namespaces configurable 2019-05-07 10:22:16 -07:00
info_test.go Masking credentials from proxy URL 2018-10-01 14:06:00 -04:00
info_unix.go Make cgroup namespaces configurable 2019-05-07 10:22:16 -07:00
info_unix_test.go Add containerd, runc, and docker-init versions to /version 2019-01-14 23:27:05 +01:00
info_windows.go Make cgroup namespaces configurable 2019-05-07 10:22:16 -07:00
inspect.go Extract volume interaction to a volumes service 2018-05-25 14:21:07 -04:00
inspect_linux.go Add canonical import comment 2018-02-05 16:51:57 -05:00
inspect_test.go Update tests to use gotest.tools 👼 2018-06-13 09:04:30 +02:00
inspect_windows.go Add canonical import comment 2018-02-05 16:51:57 -05:00
keys.go Add canonical import comment 2018-02-05 16:51:57 -05:00
keys_unsupported.go Add canonical import comment 2018-02-05 16:51:57 -05:00
kill.go Windows: Experimental: Allow containerd for runtime 2019-03-12 18:41:55 -07:00
licensing.go Expose license status in Info (#37612) 2018-08-17 17:05:21 -07:00
licensing_test.go go vet fix for TestfillLicense 2018-12-09 00:51:37 +00:00
links.go Add canonical import comment 2018-02-05 16:51:57 -05:00
list.go Fix some typos 2018-09-20 20:00:35 +08:00
list_test.go Switch to google/uuid 2019-03-13 14:13:58 +00:00
list_unix.go Add canonical import comment 2018-02-05 16:51:57 -05:00
list_windows.go Add canonical import comment 2018-02-05 16:51:57 -05:00
logdrivers_linux.go Add new local log driver 2018-08-17 09:36:56 -07:00
logdrivers_windows.go enable gcplogs driver on windows 2018-08-23 20:02:04 +00:00
logs.go daemon.ContainerLogs(): fix resource leak on follow 2018-09-06 11:47:42 -07:00
logs_test.go Add canonical import comment 2018-02-05 16:51:57 -05:00
metrics.go *: remove interfacer linter from CI 2019-03-13 11:48:39 +11:00
metrics_unix.go Fix typo: adapater -> adapter 2018-10-08 19:15:38 +08:00
metrics_unsupported.go Add canonical import comment 2018-02-05 16:51:57 -05:00
monitor.go Merge pull request #38541 from Microsoft/jjh/containerd 2019-03-19 21:09:19 -07:00
mounts.go Extract volume interaction to a volumes service 2018-05-25 14:21:07 -04:00
names.go Add canonical import comment 2018-02-05 16:51:57 -05:00
network.go Move EnableServiceDiscoveryOnDefaultNetwork to container-operations 2019-03-20 18:45:20 +01:00
nvidia_linux.go daemon: fix bug in nvidia device registration 2019-03-29 16:07:01 -07:00
oci_linux.go Make cgroup namespaces configurable 2019-05-07 10:22:16 -07:00
oci_linux_test.go oci: add integration tests for kernel.domainname configuration 2018-11-30 19:44:50 +11:00
oci_utils.go LCOW:Reworking spec builder 2019-03-12 18:41:55 -07:00
oci_windows.go Merge pull request #38541 from Microsoft/jjh/containerd 2019-03-19 21:09:19 -07:00
oci_windows_test.go Making it possible to pass Windows credential specs directly to the engine 2019-03-15 19:20:19 -07:00
pause.go Add canonical import comment 2018-02-05 16:51:57 -05:00
prune.go Move network conversions out of API router 2018-06-27 17:11:29 -07:00
reload.go Fix possible segfault in config reload 2019-01-10 15:34:02 +01:00
reload_test.go Update tests to use gotest.tools 👼 2018-06-13 09:04:30 +02:00
reload_unix.go Make cgroup namespaces configurable 2019-05-07 10:22:16 -07:00
reload_windows.go Add canonical import comment 2018-02-05 16:51:57 -05:00
rename.go docker rename enhancement 2018-09-21 09:43:06 +08:00
resize.go Windows: Experimental: Allow containerd for runtime 2019-03-12 18:41:55 -07:00
resize_test.go Update tests to use gotest.tools 👼 2018-06-13 09:04:30 +02:00
restart.go Windows: Fix restart for Hyper-V containers 2019-02-22 10:37:39 -08:00
seccomp_disabled.go Export all spec generation opts 2019-04-10 15:38:36 -04:00
seccomp_linux.go Export all spec generation opts 2019-04-10 15:38:36 -04:00
seccomp_unsupported.go Export all spec generation opts 2019-04-10 15:38:36 -04:00
secrets.go Add canonical import comment 2018-02-05 16:51:57 -05:00
secrets_linux.go Add canonical import comment 2018-02-05 16:51:57 -05:00
secrets_unsupported.go Add canonical import comment 2018-02-05 16:51:57 -05:00
secrets_windows.go Add canonical import comment 2018-02-05 16:51:57 -05:00
selinux_linux.go Add canonical import comment 2018-02-05 16:51:57 -05:00
selinux_unsupported.go Add canonical import comment 2018-02-05 16:51:57 -05:00
start.go Delete stale containerd object on start failure 2019-02-14 11:46:44 -08:00
start_unix.go vendor: update containerd to 63522d9 2018-06-08 19:19:06 -07:00
start_windows.go LCOWv1:Remote lcow.kernel and lcow.initrd 2019-03-12 19:31:12 -07:00
stats.go Switch from x/net/context -> context 2018-04-23 13:52:44 -07:00
stats_collector.go Add canonical import comment 2018-02-05 16:51:57 -05:00
stats_unix.go Add canonical import comment 2018-02-05 16:51:57 -05:00
stats_windows.go Add canonical import comment 2018-02-05 16:51:57 -05:00
stop.go daemon.ContainerStop(): fix for a negative timeout 2018-05-03 10:04:33 -07:00
top_unix.go ContainerTop: improve error message 2018-05-24 18:24:36 -07:00
top_unix_test.go Add canonical import comment 2018-02-05 16:51:57 -05:00
top_windows.go Windows: Implement docker top for containerd 2019-03-12 18:41:55 -07:00
unpause.go daemon/*.go: fix some Wrap[f]/Warn[f] errors 2018-07-11 15:51:51 +02:00
update.go Add canonical import comment 2018-02-05 16:51:57 -05:00
update_linux.go Windows: Experimental: Allow containerd for runtime 2019-03-12 18:41:55 -07:00
update_windows.go Windows: Experimental: Allow containerd for runtime 2019-03-12 18:41:55 -07:00
util_test.go Remove inmemory container map 2019-04-05 15:48:07 -04:00
uuid.go Switch to google/uuid 2019-03-13 14:13:58 +00:00
volumes.go Fix relabeling local volume source dir 2018-08-30 15:58:49 -07:00
volumes_linux.go Fix the several typos detected by github.com/client9/misspell 2018-08-09 00:45:00 +09:00
volumes_linux_test.go Use rslave propagation for mounts from daemon root 2018-02-07 14:27:09 -05:00
volumes_unit_test.go Move mount parsing to separate package. 2018-04-19 06:35:54 -04:00
volumes_unix.go daemon/mountVolumes(): eliminate MakeRPrivate call 2019-04-09 12:58:38 -07:00
volumes_unix_test.go Move mount parsing to separate package. 2018-04-19 06:35:54 -04:00
volumes_windows.go Add ADD/COPY --chown flag support to Windows 2018-08-13 21:59:11 -07:00
wait.go Switch from x/net/context -> context 2018-04-23 13:52:44 -07:00
workdir.go Add ADD/COPY --chown flag support to Windows 2018-08-13 21:59:11 -07:00