diff --git a/docs/reference/commandline/daemon.md b/docs/reference/commandline/daemon.md index 5ca00242c7..796a04c3fd 100644 --- a/docs/reference/commandline/daemon.md +++ b/docs/reference/commandline/daemon.md @@ -195,17 +195,17 @@ options for `zfs` start with `zfs`. to create and manage the thin-pool volume. This volume is then handed to Docker to exclusively create snapshot volumes needed for images and containers. - Managing the thin-pool outside of Docker makes for the most feature-rich + Managing the thin-pool outside of Engine makes for the most feature-rich method of having Docker utilize device mapper thin provisioning as the - backing storage for Docker's containers. The highlights of the lvm-based + backing storage for Docker containers. The highlights of the lvm-based thin-pool management feature include: automatic or interactive thin-pool resize support, dynamically changing thin-pool features, automatic thinp metadata checking when lvm activates the thin-pool, etc. - As a fallback if no thin pool is provided, loopback files will be + As a fallback if no thin pool is provided, loopback files are created. Loopback is very slow, but can be used without any pre-configuration of storage. It is strongly recommended that you do - not use loopback in production. Ensure your Docker daemon has a + not use loopback in production. Ensure your Engine daemon has a `--storage-opt dm.thinpooldev` argument provided. Example use: @@ -441,29 +441,33 @@ options for `zfs` start with `zfs`. * `dm.min_free_space` - Specifies the min free space percent in thin pool require for new device + Specifies the min free space percent in a thin pool require for new device creation to succeed. This check applies to both free data space as well as free metadata space. Valid values are from 0% - 99%. Value 0% disables - free space checking logic. If user does not specify a value for this optoin, - then default value for this option is 10%. + free space checking logic. If user does not specify a value for this option, + the Engine uses a default value of 10%. - Whenever a new thin pool device is created (during docker pull or - during container creation), docker will check minimum free space is - available as specified by this parameter. If that is not the case, then - device creation will fail and docker operation will fail. + Whenever a new a thin pool device is created (during `docker pull` or during + container creation), the Engine checks if the minimum free space is + available. If sufficient space is unavailable, then device creation fails + and any relevant `docker` operation fails. - One will have to create more free space in thin pool to recover from the - error. Either delete some of the images and containers from thin pool and - create free space or add more storage to thin pool. + To recover from this error, you must create more free space in the thin pool + to recover from the error. You can create free space by deleting some images + and containers from the thin pool. You can also add more storage to the thin + pool. - For lvm thin pool, one can add more storage to volume group container thin - pool and that should automatically resolve it. If loop devices are being - used, then stop docker, grow the size of loop files and restart docker and - that should resolve the issue. + To add more space to a LVM (logical volume management) thin pool, just add + more storage to the volume group container thin pool; this should automatically + resolve any errors. If your configuration uses loop devices, then stop the + Engine daemon, grow the size of loop files and restart the daemon to resolve + the issue. Example use: - $ docker daemon --storage-opt dm.min_free_space=10% + ```bash + $ docker daemon --storage-opt dm.min_free_space=10% + ``` Currently supported options of `zfs`: diff --git a/docs/userguide/storagedriver/device-mapper-driver.md b/docs/userguide/storagedriver/device-mapper-driver.md index dc5143f12f..4b7b664d1b 100644 --- a/docs/userguide/storagedriver/device-mapper-driver.md +++ b/docs/userguide/storagedriver/device-mapper-driver.md @@ -17,7 +17,9 @@ for image and container management. This article refers to the Device Mapper storage driver as `devicemapper`, and the kernel framework as `Device Mapper`. ->**Note**: The [Commercially Supported Docker Engine (CS-Engine) running on RHEL and CentOS Linux](https://www.docker.com/compatibility-maintenance) requires that you use the `devicemapper` storage driver. +>**Note**: The [Commercially Supported Docker Engine (CS-Engine) running on RHEL +and CentOS Linux](https://www.docker.com/compatibility-maintenance) requires +that you use the `devicemapper` storage driver. ## An alternative to AUFS @@ -206,103 +208,155 @@ mode uses block devices to create the thin pool. The following procedure shows you how to configure a Docker host to use the `devicemapper` storage driver in a `direct-lvm` configuration. -> **Caution:** If you have already run the Docker daemon on your Docker host +> **Caution:** If you have already run the Engine daemon on your Docker host > and have images you want to keep, `push` them Docker Hub or your private > Docker Trusted Registry before attempting this procedure. The procedure below will create a 90GB data volume and 4GB metadata volume to use as backing for the storage pool. It assumes that you have a spare block -device at `/dev/xvdf` with enough free space to complete the task. The device +device at `/dev/sdd` with enough free space to complete the task. The device identifier and volume sizes may be be different in your environment and you -should substitute your own values throughout the procedure. The procedure also -assumes that the Docker daemon is in the `stopped` state. +should substitute your own values throughout the procedure. -1. Log in to the Docker host you want to configure and stop the Docker daemon. +The procedure also assumes that the Engine daemon is in the `stopped` state. +Any existing images or data are lost by this process. -2. If it exists, delete your existing image store by removing the -`/var/lib/docker` directory. +1. Log in to the Docker host you want to configure. +2. If it is running, stop the Engine daemon. +3. Install the logical volume management version 2. - $ sudo rm -rf /var/lib/docker + ```bash + $ yum install lvm2 + ``` +4. Create a physical volume replacing `/dev/sdd` with your block device. -3. Create an LVM physical volume (PV) on your spare block device using the -`pvcreate` command. + ```bash + $ pvcreate /dev/sdd + ``` - $ sudo pvcreate /dev/xvdf - Physical volume `/dev/xvdf` successfully created +5. Create a 'docker' volume group. - The device identifier may be different on your system. Remember to -substitute your value in the command above. + ```bash + $ vgcreate docker /dev/sdd + ``` -4. Create a new volume group (VG) called `vg-docker` using the PV created in -the previous step. +6. Create a thin pool named `thinpool`. - $ sudo vgcreate vg-docker /dev/xvdf - Volume group `vg-docker` successfully created + In this example, the data logical is 95% of the 'docker' volume group size. + Leaving this free space allows for auto expanding of either the data or + metadata if space runs low as a temporary stopgap. -5. Create a new 90GB logical volume (LV) called `data` from space in the -`vg-docker` volume group. + ```bash + $ lvcreate --wipesignatures y -n thinpool docker -l 95%VG + $ lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG + ``` - $ sudo lvcreate -L 90G -n data vg-docker - Logical volume `data` created. +7. Convert the pool to a thin pool. - The command creates an LVM logical volume called `data` and an associated -block device file at `/dev/vg-docker/data`. In a later step, you instruct the -`devicemapper` storage driver to use this block device to store image and -container data. + ```bash + $ lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta + ``` - If you receive a signature detection warning, make sure you are working on -the correct devices before continuing. Signature warnings indicate that the -device you're working on is currently in use by LVM or has been used by LVM in -the past. +8. Configure autoextension of thin pools via an `lvm` profile. -6. Create a new logical volume (LV) called `metadata` from space in the -`vg-docker` volume group. + ```bash + $ vi /etc/lvm/profile/docker-thinpool.profile + ``` - $ sudo lvcreate -L 4G -n metadata vg-docker - Logical volume `metadata` created. +9. Specify 'thin_pool_autoextend_threshold' value. - This creates an LVM logical volume called `metadata` and an associated -block device file at `/dev/vg-docker/metadata`. In the next step you instruct -the `devicemapper` storage driver to use this block device to store image and -container metadata. + The value should be the percentage of space used before `lvm` attempts + to autoextend the available space (100 = disabled). -7. Start the Docker daemon with the `devicemapper` storage driver and the -`--storage-opt` flags. + ``` + thin_pool_autoextend_threshold = 80 + ``` - The `data` and `metadata` devices that you pass to the `--storage-opt` -options were created in the previous steps. +10. Modify the `thin_pool_autoextend_percent` for when thin pool autoextension occurs. - $ sudo docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata & - [1] 2163 - [root@ip-10-0-0-75 centos]# INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) - INFO[0027] Option DefaultDriver: bridge - INFO[0027] Option DefaultNetwork: bridge - - INFO[0027] Daemon has completed initialization - INFO[0027] Docker daemon commit=1b09a95-unsupported graphdriver=aufs version=1.11.0-dev + The value's setting is the perentage of space to increase the thin pool (100 = + disabled) - It is also possible to set the `--storage-driver` and `--storage-opt` flags - in the Docker config file and start the daemon normally using the `service` or - `systemd` commands. + ``` + thin_pool_autoextend_percent = 20 + ``` -8. Use the `docker info` command to verify that the daemon is using `data` and -`metadata` devices you created. +11. Check your work, your `docker-thinpool.profile` file should appear similar to the following: - $ sudo docker info - INFO[0180] GET /v1.20/info - Containers: 0 - Images: 0 - Storage Driver: devicemapper - Pool Name: docker-202:1-1032-pool - Pool Blocksize: 65.54 kB - Backing Filesystem: xfs - Data file: /dev/vg-docker/data - Metadata file: /dev/vg-docker/metadata - [...] + An example `/etc/lvm/profile/docker-thinpool.profile` file: + + ``` + activation { + thin_pool_autoextend_threshold=80 + thin_pool_autoextend_percent=20 + } + ``` + +12. Apply your new lvm profile + + ```bash + $ lvchange --metadataprofile docker-thinpool docker/thinpool + ``` + +13. Verify the `lv` is monitored. + + ```bash + $ lvs -o+seg_monitor + ``` + +14. If Engine was previously started, clear your graph driver directory. + + Clearing your graph driver removes any images and containers in your Docker + installation. + + ```bash + $ rm -rf /var/lib/docker/* + ``` + +14. Configure the Engine daemon with specific devicemapper options. + + There are two ways to do this. You can set options on the commmand line if you start the daemon there: + + ```bash + --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt dm.use_deferred_removal=true + ``` + + You can also set them for startup in the `daemon.json` configuration, for example: + + ```json + { + "storage-driver": "devicemapper", + "storage-opts": [ + "dm.thinpooldev=/dev/mapper/docker-thinpool", + "dm.use_deferred_removal=true" + ] + } + ``` +15. Start the Engine daemon. + + ```bash + $ systemctl start docker + ``` + +After you start the Engine daemon, ensure you monitor your thin pool and volume +group free space. While the volume group will auto-extend, it can still fill +up. To monitor logical volumes, use `lvs` without options or `lvs -a` to see tha +data and metadata sizes. To monitor volume group free space, use the `vgs` command. + +Logs can show the auto-extension of the thin pool when it hits the threshold, to +view the logs use: + +```bash +journalctl -fu dm-event.service +``` + +If you run into repeated problems with thin pool, you can use the +`dm.min_free_space` option to tune the Engine behavior. This value ensures that +operations fail with a warning when the free space is at or near the minimum. +For information, see the storage driver options in the Engine daemon reference. - The output of the command above shows the storage driver as `devicemapper`. - The last two lines also confirm that the correct devices are being used for -the `Data file` and the `Metadata file`. ### Examine devicemapper structures on the host @@ -336,7 +390,7 @@ Docker-MAJ:MIN-INO-pool Because Device Mapper operates at the block level it is more difficult to see diffs between image layers and containers. Docker 1.10 and later no longer -matches image layer IDs with directory names in `/var/lib/docker`. However, +matches image layer IDs with directory names in `/var/lib/docker`. However, there are two key directories. The `/var/lib/docker/devicemapper/mnt` directory contains the mount points for image and container layers. The `/var/lib/docker/devicemapper/metadata`directory contains one file for every @@ -410,3 +464,4 @@ data volumes. * [Select a storage driver](selectadriver.md) * [AUFS storage driver in practice](aufs-driver.md) * [Btrfs storage driver in practice](btrfs-driver.md) +* [daemon reference](../../reference/commandline/daemon#storage-driver-options) diff --git a/man/docker-daemon.8.md b/man/docker-daemon.8.md index cdc7781782..2b95133bdb 100644 --- a/man/docker-daemon.8.md +++ b/man/docker-daemon.8.md @@ -269,19 +269,27 @@ allocating images and container snapshots. Specifies a custom block storage device to use for the thin pool. -If using a block device for device mapper storage, it is best to use -`lvm` to create and manage the thin-pool volume. This volume is then -handed to Docker to create snapshot volumes needed for images and -containers. +If using a block device for device mapper storage, it is best to use `lvm` +to create and manage the thin-pool volume. This volume is then handed to Docker +to exclusively create snapshot volumes needed for images and containers. -Managing the thin-pool outside of Docker makes for the most feature-rich method -of having Docker utilize device mapper thin provisioning as the backing storage -for Docker's containers. The highlights of the LVM-based thin-pool management -feature include: automatic or interactive thin-pool resize support, dynamically -changing thin-pool features, automatic thinp metadata checking when lvm activates -the thin-pool, etc. +Managing the thin-pool outside of Engine makes for the most feature-rich +method of having Docker utilize device mapper thin provisioning as the +backing storage for Docker containers. The highlights of the lvm-based +thin-pool management feature include: automatic or interactive thin-pool +resize support, dynamically changing thin-pool features, automatic thinp +metadata checking when lvm activates the thin-pool, etc. -Example use: `docker daemon --storage-opt dm.thinpooldev=/dev/mapper/thin-pool` +As a fallback if no thin pool is provided, loopback files are +created. Loopback is very slow, but can be used without any +pre-configuration of storage. It is strongly recommended that you do +not use loopback in production. Ensure your Engine daemon has a +`--storage-opt dm.thinpooldev` argument provided. + +Example use: + + $ docker daemon \ + --storage-opt dm.thinpooldev=/dev/mapper/thin-pool #### dm.basesize @@ -471,27 +479,29 @@ daemon with a supported environment. #### dm.min_free_space -Specifies the min free space percent in thin pool require for new device +Specifies the min free space percent in a thin pool require for new device creation to succeed. This check applies to both free data space as well as free metadata space. Valid values are from 0% - 99%. Value 0% disables -free space checking logic. If user does not specify a value for this optoin, -then default value for this option is 10%. +free space checking logic. If user does not specify a value for this option, +the Engine uses a default value of 10%. -Whenever a new thin pool device is created (during docker pull or -during container creation), docker will check minimum free space is -available as specified by this parameter. If that is not the case, then -device creation will fail and docker operation will fail. +Whenever a new a thin pool device is created (during `docker pull` or during +container creation), the Engine checks if the minimum free space is +available. If the space is unavailable, then device creation fails and any +relevant `docker` operation fails. -One will have to create more free space in thin pool to recover from the -error. Either delete some of the images and containers from thin pool and -create free space or add more storage to thin pool. +To recover from this error, you must create more free space in the thin pool to +recover from the error. You can create free space by deleting some images +and containers from tge thin pool. You can also add +more storage to the thin pool. -For lvm thin pool, one can add more storage to volume group container thin -pool and that should automatically resolve it. If loop devices are being -used, then stop docker, grow the size of loop files and restart docker and -that should resolve the issue. +To add more space to an LVM (logical volume management) thin pool, just add +more storage to the group container thin pool; this should automatically +resolve any errors. If your configuration uses loop devices, then stop the +Engine daemon, grow the size of loop files and restart the daemon to resolve +the issue. -Example use: `docker daemon --storage-opt dm.min_free_space=10%` +Example use:: `docker daemon --storage-opt dm.min_free_space=10%` ## ZFS options