kotovalexarian-likes-github/moby--moby
1
0
Fork 0
mirror of https://github.com/moby/moby.git synced 2022-11-09 12:21:53 -05:00
Code Releases Activity

Add more detail to cluster volumes docs

Browse source 
Signed-off-by: Drew Erny <derny@mirantis.com>
This commit is contained in:
Drew Erny 2022-06-29 10:30:00 -05:00
parent 20d6b5c1bd
commit 384dc22de7
1 changed files with 144 additions and 30 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

172
docs/cluster_volumes.md
View file

@ -2,50 +2,97 @@ Cluster Volumes
=============== ===============
Docker Cluster Volumes is a new feature which allows using CSI plugins to Docker Cluster Volumes is a new feature which allows using CSI plugins to
create cluster-aware volumes create cluster-aware volumes.
The Container Storage Interface is a platform-agnostic API for storage
providers to write storage plugins which are compatible with many container
orchestrators. By leveraging the CSI, Docker Swarm can provide intelligent,
cluster-aware access to volumes across many supported storage providers.
## Installing a CSI plugin ## Installing a CSI plugin
CSI, the Container Storage Interface, defines an API for storage providers to Docker accesses CSI plugins through the Docker managed plugin system, using the
write storage plugins which are cross-compatible between various container `docker plugin` command.
orchestrators. However, most CSI plugins are shipped with configuration
specific to Kubernetes. Docker CSI Plugins use the same binaries as those for
Kubernetes, but in a different environment and sometimes with different
configuration.
If a plugin is already adapted for and available for Docker, it can be If a plugin is available for Docker, it can be installed through the `docker
installed through the `docker plugin install` command. Though such plugins may plugin install` command. Plugins may require configuration specific to the
require configuration specific to the user's environment, they will ultimately user's environment, they will ultimately be detected by and work automatically
be detected by and work automatically with Docker once enabled. with Docker once enabled.
Currently, there is no way to automatically deploy a Docker Plugin across all Currently, there is no way to automatically deploy a Docker Plugin across all
nodes in a cluster. Therefore, users must ensure the Docker Plugin is installed nodes in a cluster. Therefore, users must ensure the Docker Plugin is installed
on all nodes in the cluster on which it is desired. on all nodes in the cluster on which it is desired.
Docker Swarm worker nodes report their active plugins to the Docker Swarm The CSI plugin must be installed on all manager nodes. If a manager node does
managers, and so it is not necessary to install a plugin on every worker node not have the CSI plugin installed, a leadership change to that manager nodes
if this is not desired. However, the plugin must be installed on every manager will make Swarm unable to use that driver.
node, or a leadership change could result in Docker Swarm no longer having the
ability to call the plugin.
### Creating a Docker CSI Plugin Docker Swarm worker nodes report their active plugins to the Docker Swarm
managers, so it is not necessary to install a plugin on every worker node. The
plugin only needs to be installed on those nodes need access to the volumes
provided by that plugin.
### Multiple Instances of the Same Plugin
In some cases, it may be desirable to run multiple instances of the same
plugin. For example, there may be two different instances of some storage
provider which each need a differently configured plugin.
To run more than one instance of the same plugin, set the `--alias` option when
installing the plugin. This will cause the plugin to take a local name
different from its original name.
Ensure that when using plugin name aliases, the plugin name alias is the same
on every node.
## Creating a Docker CSI Plugin
Most CSI plugins are shipped with configuration specific to Kubernetes. They
are often provided in the form of Helm charts, and installation information may
include Kubernetes-specific steps. Docker CSI Plugins use the same binaries as
those for Kubernetes, but in a different environment and sometimes with
different configuration.
Before following this section, readers should ensure they are acquainted with Before following this section, readers should ensure they are acquainted with
the the
[Docker Engine managed plugin system](https://docs.docker.com/engine/extend/). [Docker Engine managed plugin system](https://docs.docker.com/engine/extend/).
Docker CSI plugins use this system to run. Docker CSI plugins use this system to run.
Docker Plugins consist of a root filesystem and a `config.json`. The root
filesystem can generally be exported from whatever image is built for the
plugin. The `config.json` specifies how the plugin is used. Several
CSI-specific concerns, as well as some general but poorly-documented features,
are outlined here.
### Basic Requirements
Docker CSI plugins are identified with a special interface type. There are two Docker CSI plugins are identified with a special interface type. There are two
related interfaces that CSI plugins can expose. In the `config.json`, this related interfaces that CSI plugins can expose.
should be set as such.
* `docker.csicontroller/1.0` is used for CSI Controller plugins.
* `docker.csinode/1.0` is used for CSI Node plugins.
* Combined plugins should include both interfaces.
Additionally, the interface field of the config.json includes a `socket` field.
This can be set to any value, but the CSI plugin should have its `CSI_ENDPOINT`
environment variable set appropriately.
In the `config.json`, this should be set as such:
```json ```json
"interface": { "interface": {
"types": ["docker.csicontroller/1.0","docker.csinode/1.0"] "types": ["docker.csicontroller/1.0","docker.csinode/1.0"],
} "socket": "my-csi-plugin.sock"
},
"env": [
{
"name": "CSI_ENDPOINT",
"value": "/run/docker/plugins/my-csi-plugin.sock"
}
]
``` ```
Additionally, the CSI specification states that CSI plugins should have The CSI specification states that CSI plugins should have
`CAP_SYS_ADMIN` privileges, so this should be set in the `config.json` as `CAP_SYS_ADMIN` privileges, so this should be set in the `config.json` as
well: well:
@ -55,9 +102,81 @@ well:
} }
``` ```
Other configuration is largely specific to the CSI plugin. ### Propagated Mount
#### Split-Component Plugins In order for the plugin to expose volumes to Swarm, it must publish those
volumes to a Propagated Mount location. This allows a mount to be itself
mounted to a different location in the filesystem. The Docker Plugin system
only allows one Propagated Mount, which is configured as a string representing
the path in the plugin filesystem.
When calling the CSI plugin, Docker Swarm specifies the publish target path,
which is the path in the plugin filesystem that a volume should ultimately be
used from. This is also the path that needs to be specified as the Propagated
Mount for the plugin. This path is hard-coded to be `/data/published` in the
plugin filesystem, and as such, the plugin configuration should list this as
the Propagated Mount:
```json
"propagatedMount": "/data/published"
```
### Configurable Options
Plugin configurations can specify configurable options for many fields. To
expose a field as configurable, the object including that field should include
a field `Settable`, which is an array of strings specifying the name of
settable fields.
For example, consider a plugin that supports a config file.
```json
"mounts": [
{
"name": "configfile",
"description": "Config file mounted in from the host filesystem",
"type": "bind",
"destination": "/opt/my-csi-plugin/config.yaml",
"source": "/etc/my-csi-plugin/config.yaml"
}
]
```
This configuration would result in a file located on the host filesystem at
`/etc/my-csi-plugin/config.yaml` being mounted into the plugin filesystem at
`/opt/my-csi-plugin/config.yaml`. However, hard-specifying the source path of
the configuration is undesirable. Instead, the plugin author can put the
`Source` field in the Settable array:
```json
"mounts": [
{
"name": "configfile",
"description": "Config file mounted in from the host filesystem",
"type": "bind",
"destination": "/opt/my-csi-plugin/config.yaml",
"source": "",
"settable": ["source"]
}
]
```
When a field is exposed as settable, the user can configure that field when
installing the plugin.
```
$ docker plugin install my-csi-plugin configfile.source="/srv/my-csi-plugin/config.yaml"
```
Or, alternatively, it can be set while the plugin is disabled:
```
$ docker plugin disable my-csi-plugin
$ docker plugin set my-csi-plugin configfile.source="/var/lib/my-csi-plugin/config.yaml"
$ docker plugin enable
```
### Split-Component Plugins
For split-component plugins, users can specify either the For split-component plugins, users can specify either the
`docker.csicontroller/1.0` or `docker.csinode/1.0` plugin interfaces. Manager `docker.csicontroller/1.0` or `docker.csinode/1.0` plugin interfaces. Manager
@ -198,11 +317,6 @@ Cluster Volume availability can be one of three states:
A Volume can only be removed from the cluster entirely if its availability is A Volume can only be removed from the cluster entirely if its availability is
set to `drain`, and it has been fully unpublished from all nodes. set to `drain`, and it has been fully unpublished from all nodes.
#### Force-Removing Volumes
There are cases where a Volume can get caught in a state where Swarm cannot
verify their removal. In these cases,
## Unsupported Features ## Unsupported Features
The CSI Spec allows for a large number of features which Cluster Volumes in The CSI Spec allows for a large number of features which Cluster Volumes in