2015-09-16 17:09:10 +00:00
<!-- [metadata]>
+++
title="Device mapper storage in practice"
description="Learn how to optimize your use of device mapper driver."
keywords=["container, storage, driver, device mapper"]
[menu.main]
2016-01-24 04:36:40 +00:00
parent="engine_driver"
2015-09-16 17:09:10 +00:00
+++
<![end-metadata]-->
# Docker and the Device Mapper storage driver
Device Mapper is a kernel-based framework that underpins many advanced
volume management technologies on Linux. Docker's `devicemapper` storage driver
leverages the thin provisioning and snapshotting capabilities of this framework
for image and container management. This article refers to the Device Mapper
storage driver as `devicemapper` , and the kernel framework as `Device Mapper` .
2016-04-12 19:20:03 +00:00
>**Note**: The [Commercially Supported Docker Engine (CS-Engine) running on RHEL
and CentOS Linux](https://www.docker.com/compatibility-maintenance) requires
that you use the `devicemapper` storage driver.
2015-09-16 17:09:10 +00:00
## An alternative to AUFS
Docker originally ran on Ubuntu and Debian Linux and used AUFS for its storage
backend. As Docker became popular, many of the companies that wanted to use it
were using Red Hat Enterprise Linux (RHEL). Unfortunately, because the upstream
mainline Linux kernel did not include AUFS, RHEL did not use AUFS either.
To correct this Red Hat developers investigated getting AUFS into the mainline
kernel. Ultimately, though, they decided a better idea was to develop a new
storage backend. Moreover, they would base this new storage backend on existing
`Device Mapper` technology.
Red Hat collaborated with Docker Inc. to contribute this new driver. As a result
of this collaboration, Docker's Engine was re-engineered to make the storage
backend pluggable. So it was that the `devicemapper` became the second storage
driver Docker supported.
Device Mapper has been included in the mainline Linux kernel since version
2.6.9. It is a core part of RHEL family of Linux distributions. This means that
the `devicemapper` storage driver is based on stable code that has a lot of
real-world production deployments and strong community support.
## Image layering and sharing
The `devicemapper` driver stores every image and container on its own virtual
device. These devices are thin-provisioned copy-on-write snapshot devices.
Device Mapper technology works at the block level rather than the file level.
This means that `devicemapper` storage driver's thin provisioning and
copy-on-write operations work with blocks rather than entire files.
2016-03-18 19:43:17 +00:00
>**Note**: Snapshots are also referred to as *thin devices* or *virtual
>devices*. They all mean the same thing in the context of the `devicemapper`
2016-01-11 17:39:41 +00:00
>storage driver.
2015-09-16 17:09:10 +00:00
2016-01-11 17:39:41 +00:00
With `devicemapper` the high level process for creating images is as follows:
2015-09-16 17:09:10 +00:00
1. The `devicemapper` storage driver creates a thin pool.
2016-04-05 07:35:24 +00:00
The pool is created from block devices or loop mounted sparse files (more
on this later).
2015-09-16 17:09:10 +00:00
2. Next it creates a *base device* .
2016-04-05 07:35:24 +00:00
A base device is a thin device with a filesystem. You can see which
filesystem is in use by running the `docker info` command and checking the
`Backing filesystem` value.
2015-09-16 17:09:10 +00:00
3. Each new image (and image layer) is a snapshot of this base device.
2016-04-05 07:35:24 +00:00
These are thin provisioned copy-on-write snapshots. This means that they
are initially empty and only consume space from the pool when data is written
to them.
2015-09-16 17:09:10 +00:00
2016-03-18 19:43:17 +00:00
With `devicemapper` , container layers are snapshots of the image they are
created from. Just as with images, container snapshots are thin provisioned
copy-on-write snapshots. The container snapshot stores all updates to the
container. The `devicemapper` allocates space to them on-demand from the pool
2016-01-11 17:39:41 +00:00
as and when data is written to the container.
2015-09-16 17:09:10 +00:00
2016-03-18 19:43:17 +00:00
The high level diagram below shows a thin pool with a base device and two
2016-01-11 17:39:41 +00:00
images.
2015-09-16 17:09:10 +00:00
![](images/base_device.jpg)
2016-03-18 19:43:17 +00:00
If you look closely at the diagram you'll see that it's snapshots all the way
2016-01-11 17:39:41 +00:00
down. Each image layer is a snapshot of the layer below it. The lowest layer of
2016-03-18 19:43:17 +00:00
each image is a snapshot of the base device that exists in the pool. This
2016-01-11 17:39:41 +00:00
base device is a `Device Mapper` artifact and not a Docker image layer.
2015-09-16 17:09:10 +00:00
2016-03-18 19:43:17 +00:00
A container is a snapshot of the image it is created from. The diagram below
2016-01-11 17:39:41 +00:00
shows two containers - one based on the Ubuntu image and the other based on the
Busybox image.
2015-09-16 17:09:10 +00:00
![](images/two_dm_container.jpg)
## Reads with the devicemapper
2016-03-18 19:43:17 +00:00
Let's look at how reads and writes occur using the `devicemapper` storage
driver. The diagram below shows the high level process for reading a single
2016-01-11 17:39:41 +00:00
block (`0x44f`) in an example container.
2015-09-16 17:09:10 +00:00
![](images/dm_container.jpg)
2016-01-11 17:39:41 +00:00
1. An application makes a read request for block `0x44f` in the container.
2015-09-16 17:09:10 +00:00
2016-04-05 07:35:24 +00:00
Because the container is a thin snapshot of an image it does not have the
data. Instead, it has a pointer (PTR) to where the data is stored in the image
snapshot lower down in the image stack.
2015-09-16 17:09:10 +00:00
2016-03-18 19:43:17 +00:00
2. The storage driver follows the pointer to block `0xf33` in the snapshot
2016-01-11 17:39:41 +00:00
relating to image layer `a005...` .
2015-09-16 17:09:10 +00:00
2016-03-18 19:43:17 +00:00
3. The `devicemapper` copies the contents of block `0xf33` from the image
2016-01-11 17:39:41 +00:00
snapshot to memory in the container.
2015-09-16 17:09:10 +00:00
4. The storage driver returns the data to the requesting application.
2016-04-05 07:35:24 +00:00
## Write examples
2015-09-16 17:09:10 +00:00
2016-01-11 17:39:41 +00:00
With the `devicemapper` driver, writing new data to a container is accomplished
2016-03-18 19:43:17 +00:00
by an *allocate-on-demand* operation. Updating existing data uses a
copy-on-write operation. Because Device Mapper is a block-based technology
2016-01-11 17:39:41 +00:00
these operations occur at the block level.
2015-09-16 17:09:10 +00:00
2016-03-18 19:43:17 +00:00
For example, when making a small change to a large file in a container, the
2016-01-11 17:39:41 +00:00
`devicemapper` storage driver does not copy the entire file. It only copies the
blocks to be modified. Each block is 64KB.
2015-09-16 17:09:10 +00:00
2016-04-05 07:35:24 +00:00
### Writing new data
2015-09-16 17:09:10 +00:00
To write 56KB of new data to a container:
1. An application makes a request to write 56KB of new data to the container.
2016-03-18 19:43:17 +00:00
2. The allocate-on-demand operation allocates a single new 64KB block to the
2016-01-11 17:39:41 +00:00
container's snapshot.
2015-09-16 17:09:10 +00:00
2016-04-05 07:35:24 +00:00
If the write operation is larger than 64KB, multiple new blocks are
allocated to the container's snapshot.
2015-09-16 17:09:10 +00:00
3. The data is written to the newly allocated block.
2016-04-05 07:35:24 +00:00
### Overwriting existing data
2015-09-16 17:09:10 +00:00
To modify existing data for the first time:
1. An application makes a request to modify some data in the container.
2. A copy-on-write operation locates the blocks that need updating.
2016-03-18 19:43:17 +00:00
3. The operation allocates new empty blocks to the container snapshot and
2016-01-11 17:39:41 +00:00
copies the data into those blocks.
2015-09-16 17:09:10 +00:00
4. The modified data is written into the newly allocated blocks.
The application in the container is unaware of any of these
allocate-on-demand and copy-on-write operations. However, they may add latency
to the application's read and write operations.
2016-04-05 07:35:24 +00:00
## Configure Docker with devicemapper
2015-09-16 17:09:10 +00:00
The `devicemapper` is the default Docker storage driver on some Linux
2016-03-18 19:43:17 +00:00
distributions. This includes RHEL and most of its forks. Currently, the
2016-01-11 17:39:41 +00:00
following distributions support the driver:
2015-09-16 17:09:10 +00:00
* RHEL/CentOS/Fedora
2016-03-18 19:43:17 +00:00
* Ubuntu 12.04
* Ubuntu 14.04
* Debian
2015-09-16 17:09:10 +00:00
Docker hosts running the `devicemapper` storage driver default to a
configuration mode known as `loop-lvm` . This mode uses sparse files to build
2016-03-18 19:43:17 +00:00
the thin pool used by image and container snapshots. The mode is designed to
work out-of-the-box with no additional configuration. However, production
2016-01-11 17:39:41 +00:00
deployments should not run under `loop-lvm` mode.
2015-09-16 17:09:10 +00:00
You can detect the mode by viewing the `docker info` command:
2016-04-05 07:35:24 +00:00
```bash
$ sudo docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
Pool Name: docker-202:2-25220302-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
[...]
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.93-RHEL7 (2015-01-28)
[...]
```
2015-09-16 17:09:10 +00:00
2016-03-18 19:43:17 +00:00
The output above shows a Docker host running with the `devicemapper` storage
driver operating in `loop-lvm` mode. This is indicated by the fact that the
`Data loop file` and a `Metadata loop file` are on files under
`/var/lib/docker/devicemapper/devicemapper` . These are loopback mounted sparse
2016-01-11 17:39:41 +00:00
files.
2015-09-16 17:09:10 +00:00
### Configure direct-lvm mode for production
2016-04-05 07:35:24 +00:00
The preferred configuration for production deployments is `direct-lvm` . This
2015-09-16 17:09:10 +00:00
mode uses block devices to create the thin pool. The following procedure shows
2016-03-18 19:43:17 +00:00
you how to configure a Docker host to use the `devicemapper` storage driver in
2016-01-11 17:39:41 +00:00
a `direct-lvm` configuration.
2015-09-16 17:09:10 +00:00
2016-04-05 07:35:24 +00:00
> **Caution:** If you have already run the Docker daemon on your Docker host
2016-03-18 19:43:17 +00:00
> and have images you want to keep, `push` them Docker Hub or your private
2016-01-11 17:39:41 +00:00
> Docker Trusted Registry before attempting this procedure.
2015-09-16 17:09:10 +00:00
2016-03-18 19:43:17 +00:00
The procedure below will create a 90GB data volume and 4GB metadata volume to
use as backing for the storage pool. It assumes that you have a spare block
2016-04-05 07:35:24 +00:00
device at `/dev/xvdf` with enough free space to complete the task. The device
2016-03-18 19:43:17 +00:00
identifier and volume sizes may be be different in your environment and you
2016-04-05 07:35:24 +00:00
should substitute your own values throughout the procedure. The procedure also
assumes that the Docker daemon is in the `stopped` state.
2015-09-16 17:09:10 +00:00
2016-04-05 07:35:24 +00:00
1. Log in to the Docker host you want to configure and stop the Docker daemon.
2015-09-16 17:09:10 +00:00
2016-04-05 07:35:24 +00:00
2. If it exists, delete your existing image store by removing the
`/var/lib/docker` directory.
2015-11-19 00:32:23 +00:00
2016-04-05 07:35:24 +00:00
```bash
$ sudo rm -rf /var/lib/docker
```
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
3. Create an LVM physical volume (PV) on your spare block device using the
`pvcreate` command.
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
```bash
$ sudo pvcreate /dev/xvdf
Physical volume `/dev/xvdf` successfully created
```
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
The device identifier may be different on your system. Remember to substitute
your value in the command above. If your host is running on AWS EC2, you may
need to install `lvm2` and < a href = "http://goo.gl/Q5pUwG"
target="_blank">attach an EBS device< / a > to use this procedure.
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
4. Create a new volume group (VG) called `vg-docker` using the PV created in
the previous step.
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
```bash
$ sudo vgcreate vg-docker /dev/xvdf
Volume group `vg-docker` successfully created
```
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
5. Create a new 90GB logical volume (LV) called `data` from space in the
`vg-docker` volume group.
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
```bash
$ sudo lvcreate -L 90G -n data vg-docker
Logical volume `data` created.
```
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
The command creates an LVM logical volume called `data` and an associated
block device file at `/dev/vg-docker/data` . In a later step, you instruct the
`devicemapper` storage driver to use this block device to store image and
container data.
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
If you receive a signature detection warning, make sure you are working on
the correct devices before continuing. Signature warnings indicate that the
device you're working on is currently in use by LVM or has been used by LVM in
the past.
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
6. Create a new logical volume (LV) called `metadata` from space in the
`vg-docker` volume group.
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
```bash
$ sudo lvcreate -L 4G -n metadata vg-docker
Logical volume `metadata` created.
```
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
This creates an LVM logical volume called `metadata` and an associated
block device file at `/dev/vg-docker/metadata` . In the next step you instruct
the `devicemapper` storage driver to use this block device to store image and
container metadata.
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
7. Start the Docker daemon with the `devicemapper` storage driver and the
`--storage-opt` flags.
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
The `data` and `metadata` devices that you pass to the `--storage-opt`
options were created in the previous steps.
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
```bash
$ sudo docker daemon --storage-driver=devicemapper --storage-opt dm.datadev=/dev/vg-docker/data --storage-opt dm.metadatadev=/dev/vg-docker/metadata &
[1] 2163
[root@ip-10-0-0-75 centos]# INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
INFO[0027] Option DefaultDriver: bridge
INFO[0027] Option DefaultNetwork: bridge
< -- output truncated -- >
INFO[0027] Daemon has completed initialization
INFO[0027] Docker daemon commit=1b09a95 graphdriver=aufs version=1.11.0-dev
```
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
It is also possible to set the `--storage-driver` and `--storage-opt` flags
in the Docker config file and start the daemon normally using the `service` or
`systemd` commands.
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
8. Use the `docker info` command to verify that the daemon is using `data` and
`metadata` devices you created.
2016-04-12 19:20:03 +00:00
2016-04-05 07:35:24 +00:00
```bash
$ sudo docker info
INFO[0180] GET /v1.20/info
Containers: 0
Images: 0
Storage Driver: devicemapper
Pool Name: docker-202:1-1032-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: xfs
Data file: /dev/vg-docker/data
Metadata file: /dev/vg-docker/metadata
[...]
```
2015-09-16 17:09:10 +00:00
2016-04-05 07:35:24 +00:00
The output of the command above shows the storage driver as `devicemapper` .
The last two lines also confirm that the correct devices are being used for
the `Data file` and the `Metadata file` .
2015-09-16 17:09:10 +00:00
### Examine devicemapper structures on the host
2016-03-18 19:43:17 +00:00
You can use the `lsblk` command to see the device files created above and the
2016-01-11 17:39:41 +00:00
`pool` that the `devicemapper` storage driver creates on top of them.
2015-09-16 17:09:10 +00:00
2016-04-05 07:35:24 +00:00
```bash
$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdf 202:80 0 10G 0 disk
├─vg--docker-data 253:0 0 90G 0 lvm
│ └─docker-202:1-1032-pool 253:2 0 10G 0 dm
└─vg--docker-metadata 253:1 0 4G 0 lvm
└─docker-202:1-1032-pool 253:2 0 10G 0 dm
```
2015-09-16 17:09:10 +00:00
2016-03-18 19:43:17 +00:00
The diagram below shows the image from prior examples updated with the detail
2016-01-11 17:39:41 +00:00
from the `lsblk` command above.
2015-09-16 17:09:10 +00:00
![](http://farm1.staticflickr.com/703/22116692899_0471e5e160_b.jpg)
2016-01-11 17:39:41 +00:00
In the diagram, the pool is named `Docker-202:1-1032-pool` and spans the `data`
2016-04-05 07:35:24 +00:00
and `metadata` devices created earlier. The `devicemapper` constructs the pool
name as follows:
2015-09-16 17:09:10 +00:00
```
Docker-MAJ:MIN-INO-pool
```
`MAJ` , `MIN` and `INO` refer to the major and minor device numbers and inode.
Because Device Mapper operates at the block level it is more difficult to see
2016-03-18 19:43:17 +00:00
diffs between image layers and containers. Docker 1.10 and later no longer
2016-04-12 19:20:03 +00:00
matches image layer IDs with directory names in `/var/lib/docker` . However,
2016-01-11 17:39:41 +00:00
there are two key directories. The `/var/lib/docker/devicemapper/mnt` directory
2016-03-18 19:43:17 +00:00
contains the mount points for image and container layers. The
`/var/lib/docker/devicemapper/metadata` directory contains one file for every
image layer and container snapshot. The files contain metadata about each
2016-01-11 17:39:41 +00:00
snapshot in JSON format.
2015-09-16 17:09:10 +00:00
2016-04-05 07:35:24 +00:00
## Increase capacity on a running device
You can increase the capacity of the pool on a running thin-pool device. This is
useful if the data's logical volume is full and the volume group is at full
capacity.
### For a loop-lvm configuration
2016-05-13 13:41:45 +00:00
In this scenario, the thin pool is configured to use `loop-lvm` mode. To show
the specifics of the existing configuration use `docker info` :
2016-04-05 07:35:24 +00:00
```bash
$ sudo docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 2
Server Version: 1.11.0-rc2
Storage Driver: devicemapper
Pool Name: docker-8:1-123141-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: ext4
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 1.202 GB
Data Space Total: 107.4 GB
Data Space Available: 4.506 GB
Metadata Space Used: 1.729 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.146 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.90 (2014-09-01)
Logging Driver: json-file
[...]
```
2016-04-05 07:35:24 +00:00
The `Data Space` values show that the pool is 100GB total. This example extends the pool to 200GB.
2016-04-05 07:35:24 +00:00
1. List the sizes of the devices.
```bash
$ sudo ls -lh /var/lib/docker/devicemapper/devicemapper/
2016-04-05 07:35:24 +00:00
total 1175492
-rw------- 1 root root 100G Mar 30 05:22 data
-rw------- 1 root root 2.0G Mar 31 11:17 metadata
2016-04-05 07:35:24 +00:00
```
2016-04-05 07:35:24 +00:00
2. Truncate `data` file to the size of the `metadata` file (approximage 200GB).
2016-04-05 07:35:24 +00:00
```bash
$ sudo truncate -s 214748364800 /var/lib/docker/devicemapper/devicemapper/data
```
3. Verify the file size changed.
```bash
2016-04-05 07:35:24 +00:00
$ sudo ls -al /var/lib/docker/devicemapper/devicemapper/
total 1175492
drwx------ 2 root root 4096 Mar 29 02:45 .
drwx------ 5 root root 4096 Mar 29 02:48 ..
-rw------- 1 root root 214748364800 Mar 31 11:20 data
-rw------- 1 root root 2147483648 Mar 31 11:17 metadata
2016-04-05 07:35:24 +00:00
```
4. Reload data loop device
```bash
$ sudo blockdev --getsize64 /dev/loop0
107374182400
$ sudo losetup -c /dev/loop0
$ sudo blockdev --getsize64 /dev/loop0
214748364800
```
5. Reload devicemapper thin pool.
a. Get the pool name first.
2016-05-13 13:41:45 +00:00
```bash
$ sudo dmsetup status docker-8:1-123141-pool: 0 209715200 thin-pool 91
422/524288 18338/1638400 - rw discard_passdown queue_if_no_space -
```
2016-04-05 07:35:24 +00:00
2016-05-13 13:41:45 +00:00
The name is the string before the colon.
2016-04-05 07:35:24 +00:00
2016-04-05 07:35:24 +00:00
b. Dump the device mapper table first.
2016-04-05 07:35:24 +00:00
2016-05-13 13:41:45 +00:00
```bash
$ sudo dmsetup table docker-8:1-123141-pool
0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing
```
2016-04-05 07:35:24 +00:00
c. Calculate the real total sectors of the thin pool now.
2016-05-13 13:41:45 +00:00
Change the second number of the table info (i.e. the disk end sector) to
reflect the new number of 512 byte sectors in the disk. For example, as the
new loop size is 200GB, change the second number to 419430400.
2016-04-05 07:35:24 +00:00
d. Reload the thin pool with the new sector number
2016-05-13 13:41:45 +00:00
```bash
$ sudo dmsetup suspend docker-8:1-123141-pool \
& & sudo dmsetup reload docker-8:1-123141-pool --table '0 419430400 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing' \
& & sudo dmsetup resume docker-8:1-123141-pool
```
2016-04-05 07:35:24 +00:00
#### The device_tool
The Docker's projects `contrib` directory contains not part of the core
distribution. These tools that are often useful but can also be out-of-date. < a
href="https://goo.gl/wNfDTi">In this directory, is the `device_tool.go` </ a >
which you can also resize the loop-lvm thin pool.
To use the tool, compile it first. Then, do the following to resize the pool:
```bash
$ ./device_tool resize 200GB
```
### For a direct-lvm mode configuration
In this example, you extend the capacity of a running device that uses the
2016-04-05 07:35:24 +00:00
`direct-lvm` configuration. This example assumes you are using the `/dev/sdh1`
2016-04-05 07:35:24 +00:00
disk partition.
1. Extend the volume group (VG) `vg-docker` .
```bash
$ sudo vgextend vg-docker /dev/sdh1
Volume group "vg-docker" successfully extended
```
Your volume group may use a different name.
2. Extend the `data` logical volume(LV) `vg-docker/data`
```bash
$ sudo lvextend -l+100%FREE -n vg-docker/data
Extending logical volume data to 200 GiB
Logical volume data successfully resized
```
3. Reload devicemapper thin pool.
a. Get the pool name.
2016-05-13 13:41:45 +00:00
```bash
$ sudo dmsetup status | grep pool
docker-253:17-1835016-pool: 0 96460800 thin-pool 51593 6270/1048576 701943/753600 - rw no_discard_passdown queue_if_no_space
```
2016-04-05 07:35:24 +00:00
2016-05-13 13:41:45 +00:00
The name is the string before the colon.
2016-04-05 07:35:24 +00:00
b. Dump the device mapper table.
2016-05-13 13:41:45 +00:00
```bash
$ sudo dmsetup table docker-253:17-1835016-pool
0 96460800 thin-pool 252:0 252:1 128 32768 1 skip_block_zeroing
```
2016-04-05 07:35:24 +00:00
c. Calculate the real total sectors of the thin pool now. we can use `blockdev` to get the real size of data lv.
2016-05-13 13:41:45 +00:00
Change the second number of the table info (i.e. the disk end sector) to
reflect the new number of 512 byte sectors in the disk. For example, as the
new data `lv` size is `264132100096` bytes, change the second number to
`515883008` .
2016-04-05 07:35:24 +00:00
2016-05-13 13:41:45 +00:00
```bash
$ sudo blockdev --getsize64 /dev/vg-docker/data
264132100096
```
2016-04-05 07:35:24 +00:00
d. Then reload the thin pool with the new sector number.
2016-05-13 13:41:45 +00:00
```bash
$ sudo dmsetup suspend docker-253:17-1835016-pool \
& & sudo dmsetup reload docker-253:17-1835016-pool --table '0 515883008 thin-pool 252:0 252:1 128 32768 1 skip_block_zeroing' \
& & sudo dmsetup resume docker-253:17-1835016-pool
```
2016-04-05 07:35:24 +00:00
2015-09-16 17:09:10 +00:00
## Device Mapper and Docker performance
2016-03-18 19:43:17 +00:00
It is important to understand the impact that allocate-on-demand and
2016-01-11 17:39:41 +00:00
copy-on-write operations can have on overall container performance.
2015-09-16 17:09:10 +00:00
### Allocate-on-demand performance impact
2016-03-18 19:43:17 +00:00
The `devicemapper` storage driver allocates new blocks to a container via an
allocate-on-demand operation. This means that each time an app writes to
somewhere new inside a container, one or more empty blocks has to be located
2016-01-11 17:39:41 +00:00
from the pool and mapped into the container.
2015-09-16 17:09:10 +00:00
2016-01-11 17:39:41 +00:00
All blocks are 64KB. A write that uses less than 64KB still results in a single
2016-03-18 19:43:17 +00:00
64KB block being allocated. Writing more than 64KB of data uses multiple 64KB
blocks. This can impact container performance, especially in containers that
2016-01-11 17:39:41 +00:00
perform lots of small writes. However, once a block is allocated to a container
subsequent reads and writes can operate directly on that block.
2015-09-16 17:09:10 +00:00
### Copy-on-write performance impact
2016-03-18 19:43:17 +00:00
Each time a container updates existing data for the first time, the
`devicemapper` storage driver has to perform a copy-on-write operation. This
copies the data from the image snapshot to the container's snapshot. This
2016-01-11 17:39:41 +00:00
process can have a noticeable impact on container performance.
2015-09-16 17:09:10 +00:00
2016-03-18 19:43:17 +00:00
All copy-on-write operations have a 64KB granularity. As a results, updating
32KB of a 1GB file causes the driver to copy a single 64KB block into the
container's snapshot. This has obvious performance advantages over file-level
copy-on-write operations which would require copying the entire 1GB file into
2016-01-11 17:39:41 +00:00
the container layer.
2015-09-16 17:09:10 +00:00
2016-03-18 19:43:17 +00:00
In practice, however, containers that perform lots of small block writes
2016-01-11 17:39:41 +00:00
(< 64KB ) can perform worse with `devicemapper` than with AUFS .
2015-09-16 17:09:10 +00:00
### Other device mapper performance considerations
2016-03-18 19:43:17 +00:00
There are several other things that impact the performance of the
2016-01-11 17:39:41 +00:00
`devicemapper` storage driver.
2016-03-18 19:43:17 +00:00
- **The mode.** The default mode for Docker running the `devicemapper` storage
2016-05-13 13:41:45 +00:00
driver is `loop-lvm` . This mode uses sparse files and suffers from poor
performance. It is **not recommended for production** . The recommended mode
for production environments is `direct-lvm` where the storage driver writes
directly to raw block devices.
2016-01-11 17:39:41 +00:00
- **High speed storage.** For best performance you should place the `Data file`
2016-05-13 13:41:45 +00:00
and `Metadata file` on high speed storage such as SSD. This can be direct
attached storage or from a SAN or NAS array.
2016-01-11 17:39:41 +00:00
2016-03-18 19:43:17 +00:00
- **Memory usage.** `devicemapper` is not the most memory efficient Docker
2016-05-13 13:41:45 +00:00
storage driver. Launching *n* copies of the same container loads *n* copies
of its files into memory. This can have a memory impact on your Docker host.
As a result, the `devicemapper` storage driver may not be the best choice for
PaaS and other high density use cases.
2016-01-11 17:39:41 +00:00
2016-03-18 19:43:17 +00:00
One final point, data volumes provide the best and most predictable
performance. This is because they bypass the storage driver and do not incur
any of the potential overheads introduced by thin provisioning and
copy-on-write. For this reason, you should to place heavy write workloads on
2016-01-11 17:39:41 +00:00
data volumes.
2015-09-16 17:09:10 +00:00
## Related Information
* [Understand images, containers, and storage drivers ](imagesandcontainers.md )
* [Select a storage driver ](selectadriver.md )
* [AUFS storage driver in practice ](aufs-driver.md )
2015-11-04 19:23:16 +00:00
* [Btrfs storage driver in practice ](btrfs-driver.md )
2016-04-28 06:55:22 +00:00
* [daemon reference ](../../reference/commandline/dockerd#storage-driver-options )