You are here

Configure Docker Swarm Cluster with Consul Registry service

Hi,

Here is and example of the configuration of a docker swarm cluster using the Consul software as the registry service, at the start you have some info on how to configure the docker daemon on red hat based systems that use the devicemapper storage driver,
This is a step by step config, in the next post I will put the vagrant + ansible quick install notes:


First we create 6 vms, each with 2 disks, vda for the OS, and vdb for docker-volumes and metadata.

We need a clean install docker on the hosts, and before starting the service, we use the docker-storage-setup script to create our docker storage:

IF you allready started docker stop and delete all info, be aware you will loose all your previous docker config if any:

[root@controller1 ~]# systemctl stop docker
[root@controller1 ~]# rm -Rf /var/lib/docker

the we edit the file /usr/lib/docker-storage-setup/docker-storage-setup , and leave it looking like this:

[root@controller1 ~]# cat  /usr/lib/docker-storage-setup/docker-storage-setup | grep -v ^# | sed '/^$/d'
STORAGE_DRIVER=devicemapper
DEVS=/dev/vdb
VG=dockervg
DATA_SIZE=40%FREE
MIN_DATA_SIZE=2G
CHUNK_SIZE=512K
GROWPART=false
AUTO_EXTEND_POOL=yes
POOL_AUTOEXTEND_THRESHOLD=60
POOL_AUTOEXTEND_PERCENT=20
DEVICE_WAIT_TIMEOUT=60
WIPE_SIGNATURES=false

then we can run the script and get our vg/lvols created:

[root@controller1 ~]# docker-storage-setup
INFO: Device /dev/vdb is already partitioned and is part of volume group dockervg
  Rounding up size to full physical extent 12.00 MiB
  Logical volume "docker-poolmeta" created.
  Logical volume "docker-pool" created.
  WARNING: Converting logical volume dockervg/docker-pool and dockervg/docker-poolmeta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
  Converted dockervg/docker-pool to thin pool.
  Logical volume "docker-pool" changed.

When the script runs, a new vg and lvols created, then the docker-storage config file gets modified:

[root@controller1 ~]# pvs | grep docker
  /dev/vdb1  dockervg lvm2 a--  10.00g  5.98g
[root@controller1 ~]# vgs | grep docker
  dockervg   1   1   0 wz--n- 10.00g  5.98g
[root@controller1 ~]# lvs | grep docker
  docker-poolmeta centos   -wi-a----- 32.00m                                                    
  docker-pool     dockervg twi-a-t---  3.99g             0.48   0.33                    

[root@controller1 ~]# cat /etc/sysconfig/docker-storage
DOCKER_STORAGE_OPTIONS="--storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/dockervg-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true "

Now we enable and start docker:

[root@controller1 ~]# systemctl enable docker
[root@controller1 ~]# systemctl start docker

we check all is running:

[root@controller1 ~]# docker ps && docker images
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE



So now we are going to use 3 controllers, vms controller1, controller2 and controller 3 for the docker swarm registry and swarm cluster, we are going to configure them as docker images inside our 3 hosts. 

We are going to use the internal 192.168.122.0/24 network for internal cluster managment and the 192.168.99.0/24 for the public network of the cluster.

First a quick "hello world to check docker is working fine:

[root@openstackbox ~]# ansible all -a "docker run ubuntu /bin/echo 'Hello world'"
controller3-out | SUCCESS | rc=0 >>
Hello world

controller1-out | SUCCESS | rc=0 >>
Hello world

controller2-out | SUCCESS | rc=0 >>
Hello world


At this point I have 3 servers running, because I'm going to need 3 more servers at this same point, to make life easier I'm just going to clone 3 new machines.

The config I have is a common backing file, and then 3 qcow images based on the backing file:

[root@openstackbox ~]# virsh dumpxml controller1 | grep qcow
      
[root@openstackbox ~]# qemu-img info /home/liquid/vm-nas/controller1.qcow
image: /home/liquid/vm-nas/controller1.qcow
file format: qcow2
virtual size: 30G (32212254720 bytes)
disk size: 260K
cluster_size: 65536
backing file: /home/liquid/vm-sdd/controller.qcow
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

My backing file was created just with a basic fedora install, so I'm going to commit the changes into the backing file, and the use the backing file for the 3 new servers:
This is not the best way to do this kind of task, better use some automation tool, but hey just in case it help:

[root@openstackbox ~]# qemu-img create -f qcow2 -b /home/liquid/vm-sdd/controller.qcow /home/liquid/vm-sdd/docker-host1.qcow 30G
Formatting '/home/liquid/vm-sdd/docker-host1.qcow', fmt=qcow2 size=32212254720 backing_file='/home/liquid/vm-sdd/controller.qcow' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
[root@openstackbox ~]# qemu-img create -f qcow2 -b /home/liquid/vm-sdd/controller.qcow /home/liquid/vm-sdd/docker-host2.qcow 30G
Formatting '/home/liquid/vm-sdd/docker-host2.qcow', fmt=qcow2 size=32212254720 backing_file='/home/liquid/vm-sdd/controller.qcow' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
[root@openstackbox ~]# qemu-img create -f qcow2 -b /home/liquid/vm-sdd/controller.qcow /home/liquid/vm-sdd/docker-host3.qcow 30G
Formatting '/home/liquid/vm-sdd/docker-host3.qcow', fmt=qcow2 size=32212254720 backing_file='/home/liquid/vm-sdd/controller.qcow' encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

[root@openstackbox vm-nas]# virt-clone --preserve-data -o controller1 -f /home/liquid/vm-sdd/docker-host1.qcow -f /home/liquid/vm-nas/docker-host1-data.qcow  -n docker-host1
Clone 'docker-host1' created successfully.
[root@openstackbox vm-nas]# virt-clone --preserve-data -o controller1 -f /home/liquid/vm-sdd/docker-host2.qcow -f /home/liquid/vm-nas/docker-host2-data.qcow  -n docker-host2
Clone 'docker-host2' created successfully.
[root@openstackbox vm-nas]# virt-clone --preserve-data -o controller1 -f /home/liquid/vm-sdd/docker-host3.qcow -f /home/liquid/vm-nas/docker-host3-data.qcow  -n docker-host3
Clone 'docker-host3' created successfully.

Ok, so thats another 3 hosts ready for the near future. Now lets get back on track.

Lets start the configuration of the swarm discovery service, we have 3 options to install the discovery service: consul,etc,zookeper    we are going to use the consul service.

We are going to use docker containers with and image of the consul server provided by progrium, we are going to run 1 container in each physical host:

Before we start, on the host servers  the ones that are going to be running the containers inside the swarm cluster, we need to make the docker daemon listen on a tcp port, so we can connec to the daemon from the swarm managers, by default the docker daemon on listens on a local unix socket, on a production enviroment you should secure/TLS all the comunications, between the register,swarm manager,container hosts, and cli clients, otherwise all conection info is non-encrypted and open for all to poke at.

So on nodes host1,2,3 we to get the daemon listening on port 2375 the standard non tls port, we modify the file /etc/sysconfig/docker and add the "-H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375" :

[root@host3 ~]# cat /etc/sysconfig/docker
# /etc/sysconfig/docker

# Modify these options if you want to change the way the docker daemon runs
OPTIONS='--selinux-enabled --log-driver=journald -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375'
DOCKER_CERT_PATH=/etc/docker

Restart the docker daemon and we are ready to rock.


Now on to the managment servers, where we are going to install ,the consul registry and the swarm managers:

on the first managment node we set -bootstrap-expect, so it expects 3 nodes in the cluster

[root@docker1 ~]#  docker run --restart=unless-stopped -d -h consul1 --name consul1 -v /mnt/:/data -p 192.168.124.10:8300:8300 -p 192.168.124.10:8301:8301 -p 192.168.124.10:8301:8301/udp -p 192.168.124.10:8302:8302 -p 192.168.124.10:8400:8400 -p 192.168.124.10:8500:8500 -p 192.168.123.10:53:53/udp progrium/consul -server -advertise 192.168.124.10 -bootstrap-expect 3
495e7bad5c8354495e059a52834a8775d4c9d3a1b7ec043dae944daf5af65307

And in the other 2 we use the join command:

[root@docker2 ~]# docker run --restart=unless-stopped -d -h consul2 --name consul2 -v /mnt/:/data -p 192.168.124.11:8300:8300 -p 192.168.124.11:8301:8301 -p 192.168.124.11:8301:8301/udp -p 192.168.124.11:8302:8302 -p 192.168.124.11:8400:8400 -p 192.168.124.11:8500:8500 -p 192.168.123.11:53:53/udp progrium/consul -server -advertise 192.168.124.11 -join 192.168.124.10
495e7bad5c8354495e059a52834a8775d4c9d3a1b7ec043dae944daf5af65301
[root@docker3 ~]# docker run --restart=unless-stopped -d -h consul3 --name consul3 -v /mnt/:/data -p 192.168.124.12:8300:8300 -p 192.168.124.12:8301:8301 -p 192.168.124.12:8301:8301/udp -p 192.168.124.12:8302:8302 -p 192.168.124.12:8400:8400 -p 192.168.124.12:8500:8500 -p 192.168.123.12:53:53/udp progrium/consul -server -advertise 192.168.124.12 -join 192.168.124.10
f1ac1ee6724f453456af7d06f0b2185b240239cd6b316478f7c8e053ab2a091a

We are going to check consul is running and the 3 nodes are registered in the cluster:

[root@docker1 ~]# docker exec -it consul1 /bin/bash
bash-4.3# consul members
Node     Address              Status  Type    Build  Protocol  DC
consul3  192.168.124.12:8301  alive   server  0.5.2  2         dc1
consul1  192.168.124.10:8301  alive   server  0.5.2  2         dc1
consul2  192.168.124.11:8301  alive   server  0.5.2  2         dc1

Now lets get running our swarm manager cluster, we go back to docker1 host and run

[root@docker1 ~]# docker run --restart=unless-stopped -h mgr1 --name mgr1 -d -p 3375:2375 swarm manage --replication --advertise 192.168.124.10:3375 consul://192.168.124.10:8500/
a37c024ecbcc1c525d9c2f6b7cbe72169fa5f9840453343f0a570743cb67134a
[root@docker1 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                                               NAMES
a37c024ecbcc        progrium/consul     "/bin/start -server -"   2 minutes ago       Up 2 minutes        192.168.124.10:8300-8302->8300-8302/tcp, 192.168.124.10:8400->8400/tcp, 53/tcp, 192.168.123.10:53->53/udp, 192.168.124.10:8500->8500/tcp, 192.168.124.10:8301->8301/udp, 8302/udp   consul1
c24ca28e922c        swarm               "/swarm manage --repl"   3 minutes ago       Up 3 minutes        0.0.0.0:3375->2375/tcp                                                                                                                                                              mgr1
Check the logs to see if leadership is aquired:

time="2016-09-29T08:11:39Z" level=error msg="Leader Election: watch leader channel closed, the store may be unavailable..." 
time="2016-09-29T08:11:49Z" level=info msg="Leader Election: Cluster leadership lost" 
time="2016-09-29T08:11:49Z" level=info msg="Leader Election: Cluster leadership acquired" 

We now do the same on the other nodes:

[root@docker2 ~]# docker run --restart=unless-stopped -h mgr2 --name mgr2 -d -p 3375:2375 swarm manage --replication --advertise 192.168.124.11:3375 consul://192.168.124.11:8500/
3a1a8dcbef32eb6819fd3c018c32576c346244761b396c37fb2c3064520fd404
[root@docker3 ~]# docker run --restart=unless-stopped -h mgr3 --name mgr3 -d -p 3375:2375 swarm manage --replication --advertise 192.168.124.12:3375 consul://192.168.124.12:8500
3a1a8dcbef32eb6819fd3c018c32576c346244761b396c37fb2c3064520fd405


Ok, so we have our swarm manager cluster ready, now we are going to add nodes to the swarm cluster, this nodes will contain the running docker containers.

First we are going to add a consul agent to all nodes:

[root@docker4 ~]# docker run --restart=unless-stopped -d -h consul-agt1 --name consul-agt1  -p 8300:8300 -p 8301:8301 -p 8301:8301/udp -p 8302:8302 -p 8302:8302/udp -p 8400:8400 -p 8500:8500 -p 8600:8600/udp progrium/consul -rejoin -advertise 192.168.124.13 -join 192.168.124.10
5a9ab6abc12166407aac0fb22f3df00b491cffb22bbe253016b4ca6a5479c146
[root@docker4 ~]# docker logs consul-agt1
==> WARNING: It is highly recommended to set GOMAXPROCS higher than 1
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Joining cluster...
    Join completed. Synced with 1 initial agents
==> Consul agent running!
         Node name: 'consul-agt1'
        Datacenter: 'dc1'
            Server: false (bootstrap: false)
       Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 53, RPC: 8400)
      Cluster Addr: 192.168.124.13 (LAN: 8301, WAN: 8302)
    Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
             Atlas: 
==> Log data will now stream in as it occurs:
    2016/09/30 07:49:57 [INFO] serf: EventMemberJoin: consul-agt1 192.168.124.13
    2016/09/30 07:49:57 [INFO] agent: (LAN) joining: [192.168.124.10]
    2016/09/30 07:49:57 [INFO] serf: EventMemberJoin: consul1 192.168.124.10
    2016/09/30 07:49:57 [INFO] serf: EventMemberJoin: consul3 192.168.124.12
    2016/09/30 07:49:57 [INFO] serf: EventMemberJoin: consul2 192.168.124.11
    2016/09/30 07:49:57 [INFO] agent: (LAN) joined: 1 Err: 
    2016/09/30 07:49:57 [ERR] agent: failed to sync remote state: No known Consul servers
    2016/09/30 07:49:57 [INFO] consul: adding server consul1 (Addr: 192.168.124.10:8300) (DC: dc1)
    2016/09/30 07:49:57 [INFO] consul: adding server consul3 (Addr: 192.168.124.12:8300) (DC: dc1)
    2016/09/30 07:49:57 [INFO] consul: adding server consul2 (Addr: 192.168.124.11:8300) (DC: dc1)

We just have to change the docker name and the advertise IP on the other 2 nodes:

[root@docker5 ~]# docker run --restart=unless-stopped -d -h consul-agt2 --name consul-agt2  -p 8300:8300 -p 8301:8301 -p 8301:8301/udp -p 8302:8302 -p 8302:8302/udp -p 8400:8400 -p 8500:8500 -p 8600:8600/udp progrium/consul -rejoin -advertise 192.168.124.14 -join 192.168.124.10
[root@docker6 ~]# docker run --restart=unless-stopped -d -h consul-agt3 --name consul-agt3  -p 8300:8300 -p 8301:8301 -p 8301:8301/udp -p 8302:8302 -p 8302:8302/udp -p 8400:8400 -p 8500:8500 -p 8600:8600/udp progrium/consul -rejoin -advertise 192.168.124.15 -join 192.168.124.10

The Advantage of installing a agent on the host, is that the agent is aware of all the available consul servers, and if one goes down he switches from one consul server to the other.

No lets join the nodes to the Swarm cluster, here we connect to the local consul agent we just spin up before, this way we have HA on the consul service, because the local agent has the knowledge of the cluster topology, and can switch between the diferent consul nodes.

[root@docker4 ~]# docker run -d -h join --name join swarm join --advertise 192.168.124.13:2375 consul://192.168.124.13:8500/
[root@docker5 ~]# docker run -d -h join --name join swarm join --advertise 192.168.124.14:2375 consul://192.168.124.14:8500/
[root@docker6 ~]# docker run -d -h join --name join swarm join --advertise 192.168.124.15:2375 consul://192.168.124.15:8500/

Ok, so if all went well, we have our inital work done, lets check if all looks ok:

[root@docker1 ~]# docker exec -it consul1 /bin/bash
bash-4.3# consul members
Node         Address              Status  Type    Build  Protocol  DC
consul-agt3  192.168.124.15:8301  alive   client  0.5.2  2         dc1
consul1      192.168.124.10:8301  alive   server  0.5.2  2         dc1
consul2      192.168.124.11:8301  alive   server  0.5.2  2         dc1
consul3      192.168.124.12:8301  alive   server  0.5.2  2         dc1
consu-agt1   192.168.124.13:8301  failed  client  0.5.2  2         dc1 --------> this is a typo I made, lets get rid of this client.
consul-agt1  192.168.124.13:8301  alive   client  0.5.2  2         dc1
consul-agt2  192.168.124.14:8301  alive   client  0.5.2  2         dc1

bash-4.3# consul leave consu-agt1
Graceful leave complete

bash-4.3# consul members                                                                                                                                                                              
Node         Address              Status  Type    Build  Protocol  DC
consul-agt3  192.168.124.15:8301  alive   client  0.5.2  2         dc1
consul-agt2  192.168.124.14:8301  alive   client  0.5.2  2         dc1
consul-agt1  192.168.124.13:8301  alive   client  0.5.2  2         dc1
consul2      192.168.124.11:8301  alive   server  0.5.2  2         dc1
consul3      192.168.124.12:8301  alive   server  0.5.2  2         dc1
consul1      192.168.124.10:8301  alive   server  0.5.2  2         dc1

Ok, that is better, we have our 3 servers, and 3 agents on the nodes.

We can also check info on the consul server via the rest API, check it out:

[root@docker1 ~]# curl http://192.168.124.10:8500/v1/catalog/nodes | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   338  100   338    0     0  78060      0 --:--:-- --:--:-- --:--:-- 84500
[
    {
        "Address": "192.168.124.13",
        "Node": "consu-agt1"
    },
    {
        "Address": "192.168.124.13",
        "Node": "consul-agt1"
    },
    {
        "Address": "192.168.124.14",
        "Node": "consul-agt2"
    },
    {
        "Address": "192.168.124.15",
        "Node": "consul-agt3"
    },
    {
        "Address": "192.168.124.10",
        "Node": "consul1"
    },
    {
        "Address": "192.168.124.11",
        "Node": "consul2"
    },
    {
        "Address": "192.168.124.12",
        "Node": "consul3"
    }
]

Ok, so if we check the services registered we can see there is only consul:

[root@docker1 ~]# curl http://192.168.124.10:8500/v1/catalog/services | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    13  100    13    0     0   3664      0 --:--:-- --:--:-- --:--:--  4333
{
    "consul": []
}

We need to get the registration service running so each time a new container spins up it gets registered in the cluster, for this we need another container..., we have to have it running in all nodes in the cluster:

[root@docker1 ~]# docker run -d --name registrator -h registrator -v /var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://192.168.124.10:8500
Unable to find image 'gliderlabs/registrator:latest' locally

If we now check again, the swarm cluster, has been registered:

[root@docker1 ~]# curl http://192.168.124.10:8500/v1/catalog/services | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   134  100   134    0     0  35189      0 --:--:-- --:--:-- --:--:-- 44666
{
    "consul": [],
    "consul-53": [
        "udp"
    ],
    "consul-8300": [],
    "consul-8301": [
        "udp"
    ],
    "consul-8302": [],
    "consul-8400": [],
    "consul-8500": [],
    "swarm": []
}

Lets run the registry docker in all nodes:

[root@docker2 ~]# docker run -d --name registrator -h registrator -v /var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://192.168.124.11:8500
[root@docker3 ~]# docker run -d --name registrator -h registrator -v /var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://192.168.124.12:8500
[root@docker4 ~]# docker run -d --name registrator -h registrator -v /var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://192.168.124.13:8500
[root@docker5 ~]# docker run -d --name registrator -h registrator -v /var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://192.168.124.14:8500
[root@docker6 ~]# docker run -d --name registrator -h registrator -v /var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://192.168.124.15:8500

Ok so if we drill in to the swarm service we can see our 3 swarm manager nodes:

[root@docker1 ~]# curl http://192.168.124.11:8500/v1/catalog/service/swarm | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   487  100   487    0     0   199k      0 --:--:-- --:--:-- --:--:--  237k
[
    {
        "Address": "192.168.124.10",
        "Node": "consul1",
        "ServiceAddress": "",
        "ServiceID": "registrator:mgr1:2375",
        "ServiceName": "swarm",
        "ServicePort": 3375,
        "ServiceTags": null
    },
    {
        "Address": "192.168.124.10",
        "Node": "consul1",
        "ServiceAddress": "",
        "ServiceID": "registrator:mgr2:2375",
        "ServiceName": "swarm",
        "ServicePort": 3375,
        "ServiceTags": null
    },
    {
        "Address": "192.168.124.12",
        "Node": "consul3",
        "ServiceAddress": "",
        "ServiceID": "registrator:mgr3:2375",
        "ServiceName": "swarm",
        "ServicePort": 3375,
        "ServiceTags": null
    }
]

So this is ready, lets just check to run a new container in a host, and see if it get registered:

[root@docker4 ~]# docker run -d --name web1 -p 80:80 nginx

Then we can check if it gets registered:

[root@docker1 ~]# curl http://192.168.124.11:8500/v1/catalog/services | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   175  100   175    0     0  81244      0 --:--:-- --:--:-- --:--:-- 87500
{
    "consul": [],
    "consul-53": [
        "udp"
    ],
    "consul-8300": [],
    "consul-8301": [
        "udp"
    ],
    "consul-8302": [
        "udp"
    ],
    "consul-8400": [],
    "consul-8500": [],
    "consul-8600": [
        "udp"
    ],
    "nginx-80": [],
    "swarm": []
}
[root@docker1 ~]# curl http://192.168.124.11:8500/v1/catalog/service/nginx-80 | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   166  100   166    0     0  56309      0 --:--:-- --:--:-- --:--:-- 83000
[
    {
        "Address": "192.168.124.13",
        "Node": "consul-agt1",
        "ServiceAddress": "",
        "ServiceID": "registrator:web1:80",
        "ServiceName": "nginx-80",
        "ServicePort": 80,
        "ServiceTags": null
    }
]


Great!, so the basic steps on the configuration of the swarm cluster a ready.

Now lets configure our docker CLI to connect to the swarm cluster, as easy as using the docker host variable:

[root@liquid-ibm ~]# export DOCKER_HOST=192.168.124.10:3375
[root@liquid-ibm ~]# docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      swarm/1.2.5
 API version:  1.22
 Go version:   go1.5.4
 Git commit:   27968ed
 Built:        Thu Aug 18 23:10:29 UTC 2016
 OS/Arch:      linux/amd64


[root@manager1 ~]# docker info
Containers: 9
 Running: 9
 Paused: 0
 Stopped: 0
Images: 9
Server Version: swarm/1.2.5
Role: primary
Strategy: spread
Filters: health, port, containerslots, dependency, affinity, constraint
Nodes: 3
 host1: 10.0.4.5:2375
  └ ID: VYF2:T77K:W6Y4:2HAH:C63L:NOW2:OTVA:3H6G:IYIJ:KKIH:UFAN:7EL4
  └ Status: Healthy
  └ Containers: 3 (3 Running, 0 Paused, 0 Stopped)
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.018 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.28.3.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
  └ UpdatedAt: 2016-10-02T22:07:21Z
  └ ServerVersion: 1.10.3
 host2: 10.0.5.5:2375
  └ ID: RFUE:OTN7:2JN2:GTME:B26G:SWUS:6OOZ:Y7LC:74F5:NPYS:ZTDQ:3PPS
  └ Status: Healthy
  └ Containers: 3 (3 Running, 0 Paused, 0 Stopped)
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.018 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.28.3.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
  └ UpdatedAt: 2016-10-02T22:07:01Z
  └ ServerVersion: 1.10.3
 host3: 10.0.6.5:2375
  └ ID: 4CAW:2MNR:4RHE:RZQS:KRKF:4PMY:CIDL:NX2E:2WBR:V3YG:F2YV:DBEE
  └ Status: Healthy
  └ Containers: 3 (3 Running, 0 Paused, 0 Stopped)
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 1.018 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.28.3.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
  └ UpdatedAt: 2016-10-02T22:07:11Z
  └ ServerVersion: 1.10.3
Plugins: 
 Volume: 
 Network: 
Kernel Version: 3.10.0-327.28.3.el7.x86_64
Operating System: linux
Architecture: amd64
Number of Docker Hooks: 2
CPUs: 3
Total Memory: 3.054 GiB
Name: mgr1
Registries: 


We can also check with PS all the containers running inside the cluster:

[root@manager1 ~]# docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                             NAMES
2f652bc13dcb        gliderlabs/registrator:latest   "/bin/registrator con"   15 minutes ago      Up 15 minutes                                                                                                                                                                         host1/registrator
68a1b81f93e0        gliderlabs/registrator:latest   "/bin/registrator con"   43 minutes ago      Up 12 minutes                                                                                                                                                                         host3/registrator
31506334e797        gliderlabs/registrator:latest   "/bin/registrator con"   44 minutes ago      Up 13 minutes                                                                                                                                                                         host2/registrator
082bb48e46d0        progrium/consul                 "/bin/start -rejoin -"   46 minutes ago      Up 12 minutes       10.0.6.5:8300-8302->8300-8302/tcp, 10.0.6.5:8400->8400/tcp, 10.0.6.5:8301-8302->8301-8302/udp, 53/tcp, 53/udp, 10.0.6.5:8500->8500/tcp, 10.0.6.5:8600->8600/udp   host3/consul-agt3
75e7f5c005c8        progrium/consul                 "/bin/start -rejoin -"   47 minutes ago      Up 13 minutes       10.0.5.5:8300-8302->8300-8302/tcp, 10.0.5.5:8400->8400/tcp, 10.0.5.5:8301-8302->8301-8302/udp, 53/tcp, 53/udp, 10.0.5.5:8500->8500/tcp, 10.0.5.5:8600->8600/udp   host2/consul-agt2
a2614145aa81        progrium/consul                 "/bin/start -rejoin -"   47 minutes ago      Up 16 minutes       10.0.4.5:8300-8302->8300-8302/tcp, 10.0.4.5:8400->8400/tcp, 10.0.4.5:8301-8302->8301-8302/udp, 53/tcp, 53/udp, 10.0.4.5:8500->8500/tcp, 10.0.4.5:8600->8600/udp   host1/consul-agt1


No we can test it out, lets create 6 ubuntu containers:

[root@manager1 ~]# for i in 1 2 3 4 5 6 ; do docker run -dit ubuntu /bin/bash; done
441859de3426dc549d7d725f856541308a8a53cec59a741676cec944d2885232
73bcfc6b0673033f62c169529ea84b0b90b4c6592ded6704b473379da25f9f3c
40387951fa62cc437e03936519d7ae7714117957c9902c7cf504702f61976c1f
b7f2d61ab7df316c59232116518a4540962d9dbca0d3570c2b99c0b09096c776
fec5f5670f21b369eb1836cc1b778522dcea0cb55eb8fa295183140dbb12aa33
caa4e9ad5bb2f76fe0dddf7c58b57546a3d63fcc9ff0560cbd53898e41dc441c

[root@manager1 ~]# docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED              STATUS              PORTS                                                                                                                                                             NAMES
caa4e9ad5bb2        ubuntu                          "/bin/bash"              About a minute ago   Up About a minute                                                                                                                                                                     host2/determined_euler
fec5f5670f21        ubuntu                          "/bin/bash"              About a minute ago   Up About a minute                                                                                                                                                                     host1/grave_rosalind
b7f2d61ab7df        ubuntu                          "/bin/bash"              About a minute ago   Up About a minute                                                                                                                                                                     host2/angry_pike
40387951fa62        ubuntu                          "/bin/bash"              About a minute ago   Up About a minute                                                                                                                                                                     host3/serene_yonath
73bcfc6b0673        ubuntu                          "/bin/bash"              About a minute ago   Up About a minute                                                                                                                                                                     host1/backstabbing_carson
441859de3426        ubuntu                          "/bin/bash"              About a minute ago   Up About a minute                                                                                                                                                                     host3/prickly_goldstine

Because we are using the spread strategy our containers are getting spread among all the hosts in the cluster.


So memory reservation, we can check out, how memory reservation affects the selected strategy, if not enough memory is available in the host it get removed from the filter:

[vagrant@manager1 ~]$ docker run -d -m 800m nginx
9da32ecf0e1cb63e29312f1ea222144c2095603cd4c0692e45e0d36177898364
[vagrant@manager1 ~]$ docker info | grep -E '(host|Containers|Memory)'
Containers: 10
 host1: 10.0.4.5:2375
  └ Containers: 4 (4 Running, 0 Paused, 0 Stopped)
  └ Reserved Memory: 800 MiB / 1.018 GiB
 host2: 10.0.5.5:2375
  └ Containers: 3 (3 Running, 0 Paused, 0 Stopped)
  └ Reserved Memory: 0 B / 1.018 GiB
 host3: 10.0.6.5:2375
  └ Containers: 3 (3 Running, 0 Paused, 0 Stopped)
  └ Reserved Memory: 0 B / 1.018 GiB
[vagrant@manager1 ~]$ docker run -d -m 800m nginx
c2ad2b316ec47b235d6d65e68bc6872fca493debb2ac985b4d59f22539f09a62
[vagrant@manager1 ~]$ docker info | grep -E '(host|Containers|Memory)'
Containers: 11
 host1: 10.0.4.5:2375
  └ Containers: 4 (4 Running, 0 Paused, 0 Stopped)
  └ Reserved Memory: 800 MiB / 1.018 GiB
 host2: 10.0.5.5:2375
  └ Containers: 4 (4 Running, 0 Paused, 0 Stopped)
  └ Reserved Memory: 800 MiB / 1.018 GiB
 host3: 10.0.6.5:2375
  └ Containers: 3 (3 Running, 0 Paused, 0 Stopped)
  └ Reserved Memory: 0 B / 1.018 GiB
Total Memory: 3.054 GiB

Because host1 and host 2 don't have enough space for a 250meg container, they all get scheduled to host3:


[vagrant@manager1 ~]$ docker run -d -m 250m nginx
7b28f9fbd7680e3593a6d6414ef35725d6e0037429718f18ade35ea5363dcb2c
[vagrant@manager1 ~]$ docker run -d -m 250m nginx
66eaee36612885237d25dd5742f6e427a3c7dde6566e2404a863c9caee9cb802
[vagrant@manager1 ~]$ docker run -d -m 250m nginx
00341da448307e46b3b84a96e1d64d439b2a9c93b9d98a0dce02242d97684947
[vagrant@manager1 ~]$ docker info | grep -E '(host|Containers|Memory)'
Containers: 14
 host1: 10.0.4.5:2375
  └ Containers: 4 (4 Running, 0 Paused, 0 Stopped)
  └ Reserved Memory: 800 MiB / 1.018 GiB
 host2: 10.0.5.5:2375
  └ Containers: 4 (4 Running, 0 Paused, 0 Stopped)
  └ Reserved Memory: 800 MiB / 1.018 GiB
 host3: 10.0.6.5:2375
  └ Containers: 6 (6 Running, 0 Paused, 0 Stopped)
  └ Reserved Memory: 750 MiB / 1.018 GiB
Total Memory: 3.054 GiB

Once we reach the max reservation, we get and error:

[vagrant@manager1 ~]$ docker run -d -m 250m nginx
629ec0e307717f60c34eb78c79965232f1577888898dae4421c3494480256670
[vagrant@manager1 ~]$ docker run -d -m 250m nginx
b4612a3b12b5524de470c00914c3a3406a057fdfdb5a0148320c58d964a7d569
[vagrant@manager1 ~]$ docker run -d -m 250m nginx
18b4ed1a3d98766281c07e7028cd180a1b2c3e90deff9b2bcf0ea6e3f8dd5420
[vagrant@manager1 ~]$ docker run -d -m 250m nginx
docker: Error response from daemon: no resources available to schedule container.

Lets clean up:

[vagrant@manager1 ~]$ docker ps -a | grep nginx | awk ' { print $1 }' | xargs docker rm -f 
18b4ed1a3d98
b4612a3b12b5
629ec0e30771
00341da44830
66eaee366128
7b28f9fbd768
c2ad2b316ec4
9da32ecf0e1c

Ok lets do just a quick example of affinity filters:

[vagrant@manager1 ~]$ docker run -d --name web1 nginx
f15f13cc4d936e04007fe3346ff853759f720f86e7d5b2205000966fedde0825
[vagrant@manager1 ~]$ docker ps | grep nginx
f15f13cc4d93        nginx                           "nginx -g 'daemon off"   10 seconds ago      Up 9 seconds        80/tcp, 443/tcp                                                                                                                                                   host1/web1

We created a nginx container called web1, now lets apply the "container" affinity filter passed as a variable:

[vagrant@manager1 ~]$ docker run -d -e affinity:container==web1 nginx
f84f591368c43d29b10feee5c25e2a355e5336187c6eda27fa4263a162585968
[vagrant@manager1 ~]$ docker run -d -e affinity:container==web1 nginx
b4da5269c4aa887e31d0778069fb915f97ff0a6dfd3294027b2a83bfb156451c
[vagrant@manager1 ~]$ docker ps | grep nginx
b4da5269c4aa        nginx                           "nginx -g 'daemon off"   2 seconds ago        Up 1 seconds        80/tcp, 443/tcp                                                                                                                                                   host1/naughty_turing
f84f591368c4        nginx                           "nginx -g 'daemon off"   About a minute ago   Up About a minute   80/tcp, 443/tcp                                                                                                                                                   host1/focused_bhabha
f15f13cc4d93        nginx                           "nginx -g 'daemon off"   2 minutes ago        Up 2 minutes        80/tcp, 443/tcp                                                                                                                                                   host1/web1

As you can see all the new containers are getting created in host1 where our web1 container resides, let's do a !, so it uses hosts where container web1 is NOT running:

[vagrant@manager1 ~]$ docker run -d -e affinity:container!=web1 nginx
df7d04adb2b5955e7e66af5731382f196153c0144f290607b56a7fb3727c4874
[vagrant@manager1 ~]$ docker run -d -e affinity:container!=web1 nginx
4b1546e613650741cfc3d0f379313c2c616a3c64798ce30c621b85c665b2427b
[vagrant@manager1 ~]$ docker run -d -e affinity:container!=web1 nginx
c500155ed95ad3571031449d75ced105bb4463df934b966e629e54b874c9bc7e
[vagrant@manager1 ~]$ docker run -d -e affinity:container!=web1 nginx
45f72a92aee17d7518d650039466a47643bc1c67a103494a95e7eb2faedddc33
[vagrant@manager1 ~]$ docker ps | grep nginx
45f72a92aee1        nginx                           "nginx -g 'daemon off"   2 seconds ago        Up Less than a second   80/tcp, 443/tcp                                                                                                                                                   host3/ecstatic_turing
c500155ed95a        nginx                           "nginx -g 'daemon off"   3 seconds ago        Up 2 seconds            80/tcp, 443/tcp                                                                                                                                                   host2/fervent_wing
4b1546e61365        nginx                           "nginx -g 'daemon off"   16 seconds ago       Up 14 seconds           80/tcp, 443/tcp                                                                                                                                                   host3/goofy_euclid
df7d04adb2b5        nginx                           "nginx -g 'daemon off"   18 seconds ago       Up 16 seconds           80/tcp, 443/tcp                                                                                                                                                   host2/lonely_ritchie
c07559265cae        nginx                           "nginx -g 'daemon off"   About a minute ago   Up About a minute       80/tcp, 443/tcp                                                                                                                                                   host1/web1
If the affinity we have selected can't be match the run command fails:

[vagrant@manager1 ~]$ docker run -d -e affinity:container==web1000 nginx
docker: Error response from daemon: Unable to find a node that satisfies the following conditions 
[available container slots]
[container==web1000 (soft=false)].

web1000 doesn't exist so it cant satisfy and fails, we can tell it to do a best effort and even if it can't satisfy the request, continue to run:

[vagrant@manager1 ~]$ docker run -d -e affinity:container==~web1000 nginx
fc49003257197e20cc345d20d4d03862f8486d0861b16f58ee14b2a2fd889603
[vagrant@manager1 ~]$ 


We can also use standard constraints, so we can select from the labels of the host certain label like kernel version to run the container only on that kind of host for example:

[vagrant@manager1 ~]$ docker info | grep -E '(host|Labels)'
 host1: 10.0.4.5:2375
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.28.3.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
 host2: 10.0.5.5:2375
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.28.3.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper
 host3: 10.0.6.5:2375
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.28.3.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), storagedriver=devicemapper

Here we have the same exact 3 hosts, but it could be that we have a ubuntu host, certain kernel,etc..

[vagrant@manager1 ~]$ docker run -d -e constraint:storagedriver==devicemapper nginx 
add257918ee82132a9e6dc1a21445e585d2cfdef12ab742c2e87e782cd5bbd5a

ok, lets check custom constraints, you can create labels to your nodes, this has to added on the run command of the docker daemon, so we need to modify the docker sysconfig file:

[root@host1 ~]# cat  /etc/sysconfig/docker | grep OPTION
OPTIONS='--selinux-enabled --log-driver=journald -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375i --label site=madrid --label zone=prod' 

We do the same on the other hosts and restart the docker daemon. then we check with docker info and see the new labels added:

[vagrant@manager1 ~]$ docker info | grep -E '(host|Labels)'
 host1: 10.0.4.5:2375
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.28.3.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), site=madrid, storagedriver=devicemapper, zone=prod
 host2: 10.0.5.5:2375
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.28.3.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), site=barcelona, storagedriver=devicemapper, zone=prod
 host3: 10.0.6.5:2375
  └ Labels: executiondriver=native-0.2, kernelversion=3.10.0-327.28.3.el7.x86_64, operatingsystem=CentOS Linux 7 (Core), site=gijon, storagedriver=devicemapper, zone=desarollo


So now, we can make our container run in our madrid site:

[vagrant@manager1 ~]$ docker ps | grep nginx
be8994dc8136        nginx                           "nginx -g 'daemon off"   4 seconds ago       Up 3 seconds        80/tcp, 443/tcp                                                                                                                                                   host1/nauseous_mcnulty
[vagrant@manager1 ~]$ docker run -d -e constraint:zone==desarollo nginx 
434f457f6794297ac0a2e971348daa7ce4a85fe48be0cc35d39130154dd80101
[vagrant@manager1 ~]$ docker ps | grep nginx
434f457f6794        nginx                           "nginx -g 'daemon off"   3 seconds ago       Up 2 seconds        80/tcp, 443/tcp                                                                                                                                                   host3/suspicious_pasteur
be8994dc8136        nginx                           "nginx -g 'daemon off"   19 seconds ago      Up 18 seconds       80/tcp, 443/tcp                                                                                                                                                   host1/nauseous_mcnulty
So this is clearly very powerfull...

Unix Systems: 

Add new comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.
Error | HP-UX Tips & Tricks Site

Error

Error message

  • Warning: Cannot modify header information - headers already sent by (output started at /homepages/37/d228974590/htdocs/includes/common.inc:2567) in drupal_send_headers() (line 1207 of /homepages/37/d228974590/htdocs/includes/bootstrap.inc).
  • PDOException: SQLSTATE[42000]: Syntax error or access violation: 1142 INSERT command denied to user 'dbo229817041'@'217.160.155.192' for table 'watchdog': INSERT INTO {watchdog} (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9); Array ( [:db_insert_placeholder_0] => 0 [:db_insert_placeholder_1] => cron [:db_insert_placeholder_2] => Attempting to re-run cron while it is already running. [:db_insert_placeholder_3] => a:0:{} [:db_insert_placeholder_4] => 4 [:db_insert_placeholder_5] => [:db_insert_placeholder_6] => http://www.hpuxtips.es/?q=content/configure-docker-swarm-cluster-consul-registry-service-0 [:db_insert_placeholder_7] => [:db_insert_placeholder_8] => 54.90.207.75 [:db_insert_placeholder_9] => 1512951303 ) in dblog_watchdog() (line 157 of /homepages/37/d228974590/htdocs/modules/dblog/dblog.module).
The website encountered an unexpected error. Please try again later.