Merge pull request #24857 from abronan/swarm_admin_docs
Swarm docs: add administration guide for Managers and Raft
This commit is contained in:
Sebastiaan van Stijn
GitHub
commit
e306466569
|
|
@ -0,0 +1,241 @@
|
|||
<!--[metadata]>
|
||||
+++
|
||||
aliases = [
|
||||
"/engine/swarm/manager-administration-guide/"
|
||||
]
|
||||
title = "Swarm Manager Administration Guide"
|
||||
description = "Manager administration guide"
|
||||
keywords = ["docker, container, cluster, swarm, manager, raft"]
|
||||
advisory = "rc"
|
||||
[menu.main]
|
||||
identifier="manager_admin_guide"
|
||||
parent="engine_swarm"
|
||||
weight="12"
|
||||
+++
|
||||
<![end-metadata]-->
|
||||
|
||||
# Administer and maintain a swarm of Docker Engines
|
||||
|
||||
When you run a swarm of Docker Engines, **manager nodes** are the key components
|
||||
for managing the cluster and storing the cluster state. It is important to understand
|
||||
some key features of manager nodes in order to properly deploy and maintain the
|
||||
swarm.
|
||||
|
||||
This article covers the following swarm administration tasks:
|
||||
|
||||
* [Add Manager nodes for fault tolerance](#add-manager-nodes-for-fault-tolerance)
|
||||
* [Distributing manager nodes](#distributing-manager-nodes)
|
||||
* [Running manager-only nodes](#run-manager-only-nodes)
|
||||
* [Backing up the cluster state](#back-up-the-cluster-state)
|
||||
* [Monitoring the swarm health](#monitor-swarm-health)
|
||||
* [Recovering from disaster](#recover-from-disaster)
|
||||
|
||||
Refer to [How swarm mode nodes work](how-swarm-mode-works/nodes.md)
|
||||
for a brief overview of Docker Swarm mode and the difference between manager and
|
||||
worker nodes.
|
||||
|
||||
## Operating manager nodes in a swarm
|
||||
|
||||
Swarm manager nodes use the [Raft Consensus Algorithm](raft.md) to manage the
|
||||
cluster state. You only need to understand some general concepts of Raft in
|
||||
order to manage a swarm.
|
||||
|
||||
There is no limit on the number of manager nodes. The decision about how many
|
||||
manager nodes to implement is a trade-off between performance and
|
||||
fault-tolerance. Adding manager nodes to a swarm makes the swarm more
|
||||
fault-tolerant. However, additional manager nodes reduce write performance
|
||||
because more nodes must acknowledge proposals to update the cluster state.
|
||||
This means more network round-trip traffic.
|
||||
|
||||
Raft requires a majority of managers, also called a quorum, to agree on proposed
|
||||
updates to the cluster. A quorum of managers must also agree on node additions
|
||||
and removals. Membership operations are subject to the same constraints as state
|
||||
replication.
|
||||
|
||||
## Add manager nodes for fault tolerance
|
||||
|
||||
You should maintain an odd number of managers in the swarm to support manager
|
||||
node failures. Having an odd number of managers ensures that during a network
|
||||
partition, there is a higher chance that a quorum remains available to process
|
||||
requests if the network is partitioned into two sets. Keeping a quorum is not
|
||||
guaranteed if you encounter more than two network partitions.
|
||||
|
||||
| Cluster Size | Majority | Fault Tolerance |
|
||||
|:------------:|:----------:|:-----------------:|
|
||||
| 1 | 1 | 0 |
|
||||
| 2 | 2 | 0 |
|
||||
| **3** | 2 | **1** |
|
||||
| 4 | 3 | 2 |
|
||||
| **5** | 3 | **2** |
|
||||
| 6 | 4 | 2 |
|
||||
| **7** | 4 | **3** |
|
||||
| 8 | 5 | 3 |
|
||||
| **9** | 5 | **4** |
|
||||
|
||||
For example, in a swarm with *5 nodes*, if you lose *3 nodes*, you don't have a
|
||||
quorum. Therefore you can't add or remove nodes until you recover one of the
|
||||
unavailable manager nodes or recover the cluster with disaster recovery
|
||||
commands. See [Recover from disaster](#recover-from-disaster).
|
||||
|
||||
While it is possible to scale a swarm down to a single manager node, it is
|
||||
impossible to demote the last manager node. This ensures you maintain access to
|
||||
the swarm and that the swarm can still process requests. Scaling down to a
|
||||
single manager is an unsafe operation and is not recommended. If
|
||||
the last node leaves the cluster unexpetedly during the demote operation, the
|
||||
cluster swarm will become unavailable until you reboot the node or restart with
|
||||
`--force-new-cluster`.
|
||||
|
||||
You manage cluster membership with the `docker swarm` and `docker node`
|
||||
subsystems. Refer to [Add nodes to a swarm](join-nodes.md) for more information
|
||||
on how to add worker nodes and promote a worker node to be a manager.
|
||||
|
||||
## Distributing manager nodes
|
||||
|
||||
In addition to maintaining an odd number of manager nodes, pay attention to
|
||||
datacenter topology when placing managers. For optimal fault-tolerance, distribute
|
||||
manager nodes across a minimum of 3 availability-zones to support failures of an
|
||||
entire set of machines or common maintenance scenarios. If you suffer a failure
|
||||
in any of those zones, the swarm should maintain a quorum of manager nodes
|
||||
available to process requests and rebalance workloads.
|
||||
|
||||
| Swarm manager nodes | Repartition (on 3 Availability zones) |
|
||||
|:-------------------:|:--------------------------------------:|
|
||||
| 3 | 1-1-1 |
|
||||
| 5 | 2-2-1 |
|
||||
| 7 | 3-2-2 |
|
||||
| 9 | 3-3-3 |
|
||||
|
||||
## Run manager-only nodes
|
||||
|
||||
By default manager nodes also act as a worker nodes. This means the scheduler
|
||||
can assign tasks to a manager node. For small and non-critical clusters
|
||||
assigning tasks to managers is relatively low-risk as long as you schedule
|
||||
services using **resource constraints** for *cpu* and *memory*.
|
||||
|
||||
However, because manager nodes use the Raft consensus algorithm to replicate data
|
||||
in a consistent way, they are sensitive to resource starvation. You should
|
||||
isolate managers in your swarm from processes that might block cluster
|
||||
operations like cluster heartbeat or leader elections.
|
||||
|
||||
To avoid interference with manager node operation, you can drain manager nodes
|
||||
to make them unavailable as worker nodes:
|
||||
|
||||
```bash
|
||||
docker node update --availability drain <NODE-ID>
|
||||
```
|
||||
|
||||
When you drain a node, the scheduler reassigns any tasks running on the node to
|
||||
other available worker nodes in the cluster. It also prevents the scheduler from
|
||||
assigning tasks to the node.
|
||||
|
||||
## Back up the cluster state
|
||||
|
||||
Docker manager nodes store the cluster state and manager logs in the following
|
||||
directory:
|
||||
|
||||
`/var/lib/docker/swarm/raft`
|
||||
|
||||
Back up the raft data directory often so that you can use it in case of disaster
|
||||
recovery.
|
||||
|
||||
You should never restart a manager node with the data directory from another
|
||||
node (for example, by copying the `raft` directory from one node to another).
|
||||
The data directory is unique to a node ID and a node can only use a given node
|
||||
ID once to join the swarm. (ie. Node ID space should be globally unique)
|
||||
|
||||
To cleanly re-join a manager node to a cluster:
|
||||
|
||||
1. Run `docker node demote <id-node>` to demote the node to a worker.
|
||||
2. Run `docker node rm <id-node>` before adding a node back with a fresh state.
|
||||
3. Re-join the node to the cluster using `docker swarm join`.
|
||||
|
||||
In case of [disaster recovery](#recover-from-disaster), you can take the raft data
|
||||
directory of one of the manager nodes to restore to a new swarm cluster.
|
||||
|
||||
## Monitor swarm health
|
||||
|
||||
You can monitor the health of Manager nodes by querying the docker `nodes` API
|
||||
in JSON format through the `/nodes` HTTP endpoint. Refer to the [nodes API documentation](../reference/api/docker_remote_api_v1.24.md#36-nodes)
|
||||
for more information.
|
||||
|
||||
From the command line, run `docker node inspect <id-node>` to query the nodes.
|
||||
For instance, to query the reachability of the node as a Manager:
|
||||
|
||||
```bash
|
||||
docker node inspect manager1 --format "{{ .ManagerStatus.Reachability }}"
|
||||
reachable
|
||||
```
|
||||
|
||||
To query the status of the node as a Worker that accept tasks:
|
||||
|
||||
```bash
|
||||
docker node inspect manager1 --format "{{ .Status.State }}"
|
||||
ready
|
||||
```
|
||||
|
||||
From those commands, we can see that `manager1` is both at the status
|
||||
`reachable` as a manager and `ready` as a worker.
|
||||
|
||||
An `unreachable` health status means that this particular manager node is unreachable
|
||||
from other manager nodes. In this case you need to take action to restore the unreachable
|
||||
manager:
|
||||
|
||||
- Restart the daemon and see if the manager comes back as reachable.
|
||||
- Reboot the machine.
|
||||
- If neither restarting or rebooting work, you should add another manager node or promote a worker to be a manager node. You also need to cleanly remove the failed node entry from the Manager set with `docker node demote <id-node>` and `docker node rm <id-node>`.
|
||||
|
||||
Alternatively you can also get an overview of the cluster health with `docker node ls`:
|
||||
|
||||
```bash
|
||||
# From a Manager node
|
||||
docker node ls
|
||||
ID HOSTNAME MEMBERSHIP STATUS AVAILABILITY MANAGER STATUS
|
||||
1mhtdwhvsgr3c26xxbnzdc3yp node05 Accepted Ready Active
|
||||
516pacagkqp2xc3fk9t1dhjor node02 Accepted Ready Active Reachable
|
||||
9ifojw8of78kkusuc4a6c23fx * node01 Accepted Ready Active Leader
|
||||
ax11wdpwrrb6db3mfjydscgk7 node04 Accepted Ready Active
|
||||
bb1nrq2cswhtbg4mrsqnlx1ck node03 Accepted Ready Active Reachable
|
||||
di9wxgz8dtuh9d2hn089ecqkf node06 Accepted Ready Active
|
||||
```
|
||||
|
||||
## Manager advertise address
|
||||
|
||||
When initiating or joining a Swarm cluster, you have to specify the `--listen-addr`
|
||||
flag to advertise your address to other Manager nodes in the cluster.
|
||||
|
||||
We recommend that you use a *fixed IP address* for the advertised address, otherwise
|
||||
the cluster could become unstable on machine reboot.
|
||||
|
||||
Indeed if the whole cluster restarts and every Manager gets a new IP address on
|
||||
restart, there is no way for any of those nodes to contact an existing Manager
|
||||
and the cluster will stay stuck trying to contact other nodes through their old address.
|
||||
While having dynamic IP addresses for Worker nodes is acceptable, Managers are
|
||||
meant to be a stable piece in the infrastructure thus it is highly recommended to
|
||||
deploy those critical nodes with static IPs.
|
||||
|
||||
## Recover from disaster
|
||||
|
||||
Swarm is resilient to failures and the cluster can recover from any number
|
||||
of temporary node failures (machine reboots or crash with restart).
|
||||
|
||||
In a swarm of `N` managers, there must be a quorum of manager nodes greater than
|
||||
50% of the total number of managers (or `(N/2)+1`) in order for the swarm to
|
||||
process requests and remain available. This means the swarm can tolerate up to
|
||||
`(N-1)/2` permanent failures beyond which requests involving cluster management
|
||||
cannot be processed. These types of failures include data corruption or hardware
|
||||
failures.
|
||||
|
||||
Even if you follow the guidelines here, it is possible that you can lose a
|
||||
quorum of manager nodes. If you can't recover the quorum by conventional
|
||||
means such as restarting faulty nodes, you can recover the cluster by running
|
||||
`docker swarm init --force-new-cluster` on a manager node.
|
||||
|
||||
```bash
|
||||
# From the node to recover
|
||||
docker swarm init --force-new-cluster --listen-addr node01:2377
|
||||
```
|
||||
|
||||
The `--force-new-cluster` flag puts the Docker Engine into swarm mode as a
|
||||
manager node of a single-node cluster. It discards cluster membership information
|
||||
that existed before the loss of the quorum but it retains data necessary to the
|
||||
Swarm cluster such as services, tasks and the list of worker nodes.
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
<!--[metadata]>
|
||||
+++
|
||||
title = "Raft consensus in swarm mode"
|
||||
description = "Raft consensus algorithm in swarm mode"
|
||||
keywords = ["docker, container, cluster, swarm, raft"]
|
||||
advisory = "rc"
|
||||
[menu.main]
|
||||
identifier="raft"
|
||||
parent="engine_swarm"
|
||||
weight="13"
|
||||
+++
|
||||
<![end-metadata]-->
|
||||
|
||||
## Raft consensus algorithm
|
||||
|
||||
When the Docker Engine runs in swarm mode, manager nodes implement the
|
||||
[Raft Consensus Algorithm](http://thesecretlivesofdata.com/raft/) to manage the global cluster state.
|
||||
|
||||
The reason why *Docker swarm mode* is using a consensus algorithm is to make sure that
|
||||
all the manager nodes that are in charge of managing and scheduling tasks in the cluster,
|
||||
are storing the same consistent state.
|
||||
|
||||
Having the same consistent state across the cluster means that in case of a failure,
|
||||
any Manager node can pick up the tasks and restore the services to a stable state.
|
||||
For example, if the *Leader Manager* which is responsible for scheduling tasks in the
|
||||
cluster dies unexpectedly, any other Manager can pick up the task of scheduling and
|
||||
re-balance tasks to match the desired state.
|
||||
|
||||
Systems using consensus algorithms to replicate logs in a distributed systems
|
||||
do require special care. They ensure that the cluster state stays consistent
|
||||
in the presence of failures by requiring a majority of nodes to agree on values.
|
||||
|
||||
Raft tolerates up to `(N-1)/2` failures and requires a majority or quorum of
|
||||
`(N/2)+1` members to agree on values proposed to the cluster. This means that in
|
||||
a cluster of 5 Managers running Raft, if 3 nodes are unavailable, the system
|
||||
will not process any more requests to schedule additional tasks. The existing
|
||||
tasks will keep running but the scheduler will not be able to rebalance tasks to
|
||||
cope with failures if when the manager set is not healthy.
|
||||
|
||||
The implementation of the consensus algorithm in swarm mode means it features
|
||||
the properties inherent to distributed systems:
|
||||
|
||||
- *agreement on values* in a fault tolerant system. (Refer to [FLP impossibility theorem](http://the-paper-trail.org/blog/a-brief-tour-of-flp-impossibility/)
|
||||
and the [Raft Consensus Algorithm paper](https://www.usenix.org/system/files/conference/atc14/atc14-paper-ongaro.pdf))
|
||||
- *mutual exclusion* through the leader election process
|
||||
- *cluster membership* management
|
||||
- *globally consistent object sequencing* and CAS (compare-and-swap) primitives
|
||||
Loading…
Reference in New Issue