Main menu
You are here
Continuous Delivery Workflow example with Docker, Docker-Compose and Ansible part 3
Mon, 10/31/2016 - 11:22 — liquid
Hi,
Previous Posts related to this one:
http://hpuxtips.es/?q=content/continuous-delivery-workflow-example-docke...
http://hpuxtips.es/?q=content/continuous-delivery-workflow-example-docke...
http://hpuxtips.es/?q=content/docker-using-docker-compose-build-lamp-dev...
Git repo with all the files we work with in these posts:
https://github.com/likid0/CDdockerAnsible
In this post we are going to automate inside our Makefile the tagging and pushing of our application release image to dockerhub.
First we are going to publish our images to dockerhub, we are going to publish 3 images:
1. the liquidapp image with our application [Manually Built]
2. the ansible agent image [AutoBuild]
3. the test release image. [AutoBuild]
the liquidapp is going to be configured for manual builds, and the ansible and testrel images are going to be configured with the auto-build feature of docker hub.
So first we build our image with our company/user in github:
[liquid@liquid-ibm:lamp-docker/liquid-app]$ docker build -t likid0/liquidapp -f docker/release/Dockerfile . (10-28 11:46)
Sending build context to Docker daemon 58.37 kB
Step 1 : FROM eboraas/apache-php
---> eb51da88ee48
Step 2 : MAINTAINER Daniel Alexander Parkes
---> Using cache
---> 4fa486f31c1a
Step 3 : LABEL version "0.2"
---> Using cache
---> fb589b85cfac
Step 4 : RUN echo 'export DBNAME=${DBNAME}\nexport DBUSER=${DBUSER}\nexport DBPASS=${DBPASS}\nexport DBHOST=${DBHOST}\n' >> /etc/apache2/envvars
---> Using cache
---> e9265f57344e
Step 5 : RUN sed -i -e 's/DirectoryIndex.*$/DirectoryIndex index.php/g' /etc/apache2/mods-available/dir.conf && rm /etc/apache2/sites-enabled/default-ssl.conf
---> Using cache
---> 18f55f7d78d7
Step 6 : EXPOSE 80
---> Using cache
---> 5671069b8619
Step 7 : COPY src/index.php /var/www/html/
---> 0a689c90b5a9
Removing intermediate container e75981936e0a
Step 8 : CMD /usr/sbin/apache2ctl -D FOREGROUND
---> Running in a8f940f905db
---> 2ef93e0c36f6
Removing intermediate container a8f940f905db
Successfully built 2ef93e0c36f6
We need to login to docker hub to push and image to docker.io:
[liquid@liquid-ibm:lamp-docker/liquid-app]$ docker login (10-28 11:48) Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username: likid0 Password: Login Succeeded
In this case we are pushing the image to docker.io so there is no need to specify the "docker.io" because it's de default
If we wanted to push to another registry we would have to specify it for example gitlab.com/likid0/liquidapp
[liquid@liquid-ibm:lamp-docker/liquid-app]$ docker push likid0/liquidapp (10-28 11:49) The push refers to a repository [docker.io/likid0/liquidapp] fce96d0580f3: Pushed a22f0f21b9a2: Pushed 762ba8b2d1a0: Pushed 4d754c8ed4ab: Pushed 68e70936b56e: Pushed c31e0f7dff0c: Pushed e6d9a7cdfe75: Pushed 8a4b898be86d: Pushed 2fa9134afa1b: Pushed ce18d5c32c0b: Pushed 30b04bbd76f8: Pushed e2fb921843ab: Pushed e306a95461ac: Pushed latest: digest: sha256:15c507412ece1be5d5ae56db1654b85b5e0a755f3002d1ee08f863ebc35f7cc7 size: 3029
We have configured 2 autobuild images in dockerhub, we are not going to go into details on how to do this part, you just need to link your github account to dockerhub
And in DockerHub point to your github repo that contains the Dockerfile of the Image you want to autobuild.
In our case is the ansible and reltest images, we continue with the autobuild for the 2 images configured:
[liquid@liquid-ibm:liquid-app/docker]$ docker search likid0 (10-28 12:31) NAME DESCRIPTION STARS OFFICIAL AUTOMATED likid0/ansibledockerimage Ansible Docker Image for probing tcp ports 0 [OK] likid0/reltestdockerimage Relelease Test Container image to test liq... 0 [OK] likid0/liquidapp Liquidapp it's a useless php image just fo... 0 Let's test the autobuild feature, first I pull the ansible image: [liquid@liquid-ibm:lamp-docker/ansible]$ docker pull likid0/ansibledockerimage (10-28 12:50) Using default tag: latest latest: Pulling from likid0/ansibledockerimage bf5d46315322: Already exists 9f13e0ac480c: Already exists e8988b5b3097: Already exists 40af181810e7: Already exists ff283f7fb8e4: Pull complete 249f7b790b3e: Pull complete Digest: sha256:745429b124360072e4724f95038f687a443d6bf32c45370a0203a5a3222f7861 Status: Downloaded newer image for likid0/ansibledockerimage:latest [liquid@liquid-ibm:lamp-docker/ansible]$ docker pull likid0/ansibledockerimage (10-28 12:50) Using default tag: latest latest: Pulling from likid0/ansibledockerimage bf5d46315322: Already exists 9f13e0ac480c: Already exists e8988b5b3097: Already exists 40af181810e7: Already exists ff283f7fb8e4: Pull complete 249f7b790b3e: Pull complete Digest: sha256:745429b124360072e4724f95038f687a443d6bf32c45370a0203a5a3222f7861 Status: Downloaded newer image for likid0/ansibledockerimage:latest [liquid@liquid-ibm:lamp-docker/ansible]$ docker run likid0/ansibledockerimage | grep Waiting (10-28 13:00) TASK [Waiting for DB] **********************************************************
Then I modify a line of code to fix the -name definition on the ansible playbook:
[liquid@liquid-ibm:lamp-docker/ansible]$ vi probe-service.yml (10-28 12:49)
[liquid@liquid-ibm:lamp-docker/ansible]$ git commit . (10-28 12:50)
[master f6c08af] modified -name to remove db for service in generic as it can probe any port/service
1 file changed, 1 insertion(+), 1 deletion(-)
[liquid@liquid-ibm:lamp-docker/ansible]$ git diff 15091572dce9c1f2a1245f6679770631ef25e85b (10-28 12:57)
diff --git a/probe-service.yml b/probe-service.yml
index 91d80cb..49054d6 100644
--- a/probe-service.yml
+++ b/probe-service.yml
@@ -10,7 +10,7 @@
probe_port: "{{ lookup('env','PROBE_PORT') }}"
probe_delay: "{{ lookup('env','PROBE_DELAY') | default(0, true) }}"
probe_timeout: "{{ lookup('env','PROBE_TIMEOUT') | default (80, true) }}"
- - name: Waiting for DB
+ - name: Waiting for TCP service to be available
Now I push the changes to github, that should start a new build of the ansible image:
[liquid@liquid-ibm:lamp-docker/ansible]$ git push -u origin master (10-28 12:51) Username for 'https://github.com': likid0 Password for 'https://likid0@github.com': Counting objects: 3, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 393 bytes | 0 bytes/s, done. Total 3 (delta 1), reused 0 (delta 0) remote: Resolving deltas: 100% (1/1), completed with 1 local objects. To https://github.com/likid0/AnsibleDockerImage.git 1509157..f6c08af master -> master Branch master set up to track remote branch master from origin. After 10 minutes we pull the image, and we are going to check our changes are there: [liquid@liquid-ibm:lamp-docker/ansible]$ docker pull likid0/ansibledockerimage (10-28 13:01) Using default tag: latest latest: Pulling from likid0/ansibledockerimage bf5d46315322: Already exists 9f13e0ac480c: Already exists e8988b5b3097: Already exists 40af181810e7: Already exists 99f779855b4e: Pull complete 78fa1ee1964d: Pull complete Digest: sha256:3bbaba14740d6a0bead413512d833c461c8ee722e2e80278ebce257daa8bb7d3 Status: Downloaded newer image for likid0/ansibledockerimage:latest [liquid@liquid-ibm:lamp-docker/ansible]$ docker run likid0/ansibledockerimage | grep Waiting (10-28 13:01) TASK [Waiting for TCP service to be available] *********************************
Ok autobuild is working ok, lets modify in our docker compose file, the reference to the images.
Because now we wan't to use the dockerhub images, not our previous local images:
[liquid@liquid-ibm:liquid-app/docker]$ cat dev/docker-compose.yml | grep -i image (10-28 12:31)
image: mysql:5.6
image: likid0/ansibledockerimage
[liquid@liquid-ibm:liquid-app/docker]$ cat release/docker-compose.yml | grep -i image (10-28 12:31)
image: mysql:5.6
image: dockercloud/haproxy
image: likid0/ansibledockerimage
image: likid0/ansibledockerimage
image: likid0/reltestdockerimage
So once we have our liquidapp image in docker hub, we want automate the push/publish process in our make file.
So after we run our test and release stages we can automaticly deploy the new image to docker hub.
We add to our Makefile:
IMAGE_NAME ?= liquidapp
DEV_PRO := liquidev
REL_PRO := liquidrel
DEV_FILE := docker/dev/docker-compose.yml
REL_FILE := docker/release/docker-compose.yml
#Here we add variables, to use with our new stages:
#Docker registry we are using the default docker.io
DOCKER_REGISTRY ?= docker.io
PROJECT_NAME ?= liquidapp
#ORG_NAME our user name in docker hub
ORG_NAME ?= likid0
#The service name inside the compose file that contains the app with the image we want to publish
APP_SERVICE_NAME := web
#Name of the image we publish in docker hub
IMAGE_NAME ?= liquidapp
#We add our new stages
.PHONY: devtest build release clean login logout publish
devtest:
@ echo "Pulling latest images..."
@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) pull db
@ echo "Building images..."
@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) build --pull web
@ echo "Ensuring database is ready..."
@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) run --rm agent
@ echo "Running tests..."
@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) up web
release:
@ echo "Starting release env"
@ echo "Building images..."
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) build web
@ echo "Ensuring database is ready..."
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) run --rm agent-db
@ echo "Starting release env services"
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) up -d lb
@ echo "Ensuring HAproxy is ready..."
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) run --rm agent-db
@ echo "Scaling web services"
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) scale web=3
@ echo "Running tests..."
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) run --rm test
clean:
@ echo "Destroying dev env"
@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) kill
@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) rm --all -f -v
@ echo "Destroying Release env"
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) kill
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) rm --all -f -v
@ echo "Removing dangling objects"
@ docker images -q -f dangling=true -f label=application=$(IMAGE_NAME) | xargs -I ARGS docker rmi -f ARGS
@ echo "Clean complete"
# here we do a docker login, we are going to use shell variables so we don't hardwire the pass in the make file
login:
@ echo "Logging in to Docker registry $$DOCKER_REGISTRY..."
@ docker login -u $$DOCKER_USER -p $$DOCKER_PASSWORD $(DOCKER_REGISTRY_AUTH)
@ echo "Logged in to Docker registry $$DOCKER_REGISTRY"
# logout is and easy one :D
logout:
@ echo "Logging out of Docker registry $$DOCKER_REGISTRY..."
@ docker logout
@ echo "Logged out of Docker registry $$DOCKER_REGISTRY"
Ok lets test our login and logout:
[liquid@liquid-ibm:lamp-docker/liquid-app]$ export DOCKER_USER=likid0 (10-31 11:08)
[liquid@liquid-ibm:lamp-docker/liquid-app]$ export DOCKER_PASSWORD=XXXXXXX (10-31 11:08)
[liquid@liquid-ibm:lamp-docker/liquid-app]$ make login (10-31 11:08)
Logging in to Docker registry ...
Login Succeeded
Logged in to Docker registry
[liquid@liquid-ibm:lamp-docker/liquid-app]$ make logout (10-31 11:08)
Logging out of Docker registry ...
Remove login credentials for https://index.docker.io/v1/
Logged out of Docker registry
Ok login and logout is working great, now we wan't to tag and publish our images.
So we edit our make file again:
#Here we add variables, to use with our new stages:
#Docker registry we are using the default docker.io
DOCKER_REGISTRY ?= docker.io
PROJECT_NAME ?= liquidapp
#ORG_NAME our user name in docker hub
ORG_NAME ?= likid0
#The service name inside the compose file that contains the app with the image we want to publish
APP_SERVICE_NAME := web
#Name of the image we publish in docker hub
IMAGE_NAME ?= liquidapp
#We add our new stages
.PHONY: devtest build release clean login logout publish
devtest:
@ echo "Pulling latest images..."
@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) pull db
@ echo "Building images..."
@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) build --pull web
@ echo "Ensuring database is ready..."
@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) run --rm agent
@ echo "Running tests..."
@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) up web
release:
@ echo "Starting release env"
@ echo "Building images..."
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) build web
@ echo "Ensuring database is ready..."
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) run --rm agent-db
@ echo "Starting release env services"
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) up -d lb
@ echo "Ensuring HAproxy is ready..."
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) run --rm agent-db
@ echo "Scaling web services"
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) scale web=3
@ echo "Running tests..."
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) run --rm test
clean:
@ echo "Destroying dev env"
@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) kill
@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) rm --all -f -v
@ echo "Destroying Release env"
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) kill
@ docker-compose -p $(REL_PRO) -f $(REL_FILE) rm --all -f -v
@ echo "Removing dangling objects"
@ docker images -q -f dangling=true -f label=application=$(IMAGE_NAME) | xargs -I ARGS docker rmi -f ARGS
@ echo "Clean complete"
# here we do a docker login, we are going to use shell variables so we don't hardwire the pass in the make file
login:
@ echo "Logging in to Docker registry $$DOCKER_REGISTRY..."
@ docker login -u $$DOCKER_USER -p $$DOCKER_PASSWORD $(DOCKER_REGISTRY_AUTH)
@ echo "Logged in to Docker registry $$DOCKER_REGISTRY"
# logout is and easy one :D
logout:
@ echo "Logging out of Docker registry $$DOCKER_REGISTRY..."
@ docker logout
@ echo "Logged out of Docker registry $$DOCKER_REGISTRY"
#Here we want to be able to pass as arguments the tags we want to add to our image when pushing to dockerhub
#So for example we would run
#make 0.2 latest
#and we would get our liquidapp image tagged with 0.1 and latest like:
#docker.io/likid0/liquidapp:latest and docker.io/likid0/liquidapp:0.1
tag:
@echo "Tagging release image with tags $(TAG_ARGS)..."
#here we use a for, where for each tag_args variable that get's passed we use docker tag with the image id to tag it
@ $(foreach tag,$(TAG_ARGS), docker tag $(IMAGE_ID) $(DOCKER_REGISTRY)/$(ORG_NAME)/$(IMAGE_NAME):$(tag);)
@echo "Tagging complete"
#Here we do a push to docker hub with all the tags we generated in the tag stage
publish:
@echo "Publishing release image $(IMAGE_ID) to $(DOCKER_REGISTRY)/$(ORG_NAME)/$(IMAGE_NAME)..."
@ $(foreach tag,$(shell echo $(REPO_EXPR)), docker push $(tag);)
@echo "Publish complete"
# Repository Filter, so we only select images that are from our organization/User
REPO_FILTER := $(ORG_NAME)/$(IMAGE_NAME)[^[:space:]|\$$]*
# Here we get all the tags that our images is currently tagged with, so we can publish them:
REPO_EXPR := $$(docker inspect -f '{{range .RepoTags}}{{.}} {{end}}' $(IMAGE_ID) | grep -oh "$(REPO_FILTER)" | xargs)
#Get container id of application service container
#HEre we get the app container id that is running the application in our example is the web service
APP_CONTAINER_ID := $$(docker-compose -p $(REL_PRO) -f $(REL_FILE) ps -q $(APP_SERVICE_NAME) | head -1)
#Get image id of application service
#With the APP_CONTAINER_ID we get from the previous command we extrac the Image id of that container
#Now we have the image id that we wan't to tag and later publish to dockerhub
IMAGE_ID := $$(docker inspect -f '{{ .Image }}' $(APP_CONTAINER_ID))
# Extract tag arguments
# here we use the internal $(MAKECMDGOALS) variable, that get's all the args passed to the make command
ifeq (tag,$(firstword $(MAKECMDGOALS)))
# we extract all the args into tag_args
TAG_ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
ifeq ($(TAG_ARGS),)
$(error You must specify a tag)
endif
$(eval $(TAG_ARGS):;@:)
endif
Just running the commands we use in our make file manually from the cli, so its clearer how they work:
[liquid@liquid-ibm:docker/release]$ docker-compose -p liquidrel ps (10-31 11:30)
Name Command State Ports
------------------------------------------------------------------------------------------------
liquidrel_db_1 docker-entrypoint.sh mysqld Up 3306/tcp
liquidrel_lb_1 /sbin/tini -- dockercloud- ... Up 1936/tcp, 443/tcp, 0.0.0.0:80->80/tcp
liquidrel_web_1 /usr/sbin/apache2ctl -D FO ... Up 443/tcp, 80/tcp
liquidrel_web_2 /usr/sbin/apache2ctl -D FO ... Up 443/tcp, 80/tcp
liquidrel_web_3 /usr/sbin/apache2ctl -D FO ... Up 443/tcp, 80/tcp
Here we get the container ID of the web service:
[liquid@liquid-ibm:docker/release]$ docker-compose -p liquidrel ps -q web | head -1 (10-31 11:31)
6be624a259cee315578ddadacc98ce745d579217064fb0316164cfedd9eb7f44
With the Container ID of the web service, we extract the Image ID of the image we wan't to push:
[liquid@liquid-ibm:docker/release]$ docker inspect -f '{{ .Image }}' 6be624a259cee315578ddadacc98ce745d579217064fb0316164cfedd9eb7f44 (10-31 11:31)
sha256:0c3ca82c84b2ba76a6028551e4329702a64351af4e0f46ed0aa070150b2edd5d
And to get the REPO_EXPR,example:
REPO_EXPR := $$(docker inspect -f '{{range .RepoTags}}{{.}} {{end}}' $(IMAGE_ID) | grep -oh "$(REPO_FILTER)" | xargs)
We use the docker inspect agains the image id and get a list of all the tags for that image:
[liquid@liquid-ibm:docker/release]$ docker inspect -f '{{range .RepoTags}}{{.}} {{end}}' sha256:0c3ca82c84b2ba76a6028551e4329702a64351af4e0f46ed0aa070150b2edd5d
likid0/liquidapp:0.1 likid0/liquidapp:0.2 likid0/liquidapp:latest liquidrel_web:latest
The we apply the REPO_FILTER
[liquid@liquid-ibm:docker/release]$ docker inspect -f '{{range .RepoTags}}{{.}} {{end}}' sha256:0c3ca82c84b2ba76a6028551e4329702a64351af4e0f46ed0aa070150b2edd5d | grep "likid0/liquidapp[^[:space:]|\$$]*" | xargs
likid0/liquidapp:0.1 likid0/liquidapp:0.2 likid0/liquidapp:latest
Ok, so with the new tag and publish stages ready lets test them out:
liquid@liquid-ibm:lamp-docker/liquid-app]$ make devtest
liquid@liquid-ibm:lamp-docker/liquid-app]$ make release
[liquid@liquid-ibm:lamp-docker/liquid-app]$ make tag 0.2 latest (10-31 11:21)
Tagging release image with tags 0.2 latest...
Tagging complete
[liquid@liquid-ibm:lamp-docker/liquid-app]$ docker images likid0/liquidapp (10-31 11:21)
REPOSITORY TAG IMAGE ID CREATED SIZE
likid0/liquidapp 0.2 0c3ca82c84b2 34 minutes ago 275.7 MB
likid0/liquidapp latest 0c3ca82c84b2 34 minutes ago 275.7 MB
[liquid@liquid-ibm:lamp-docker/liquid-app]$ make login (10-31 11:22)
Logging in to Docker registry ...
Login Succeeded
Logged in to Docker registry
[liquid@liquid-ibm:lamp-docker/liquid-app]$ make publish (10-31 12:09)
Publishing release image sha256:4238f1b341fb83822ece89fbd19428f2c1a6860defc19f46e1eb7f4182f6d926 to docker.io/likid0/liquidapp...
The push refers to a repository [docker.io/likid0/liquidapp]
86f91eb1cf0d: Pushed
6eac29325f1a: Pushed
06f22bc3a995: Pushed
d2223eb091a9: Mounted from eboraas/apache-php
876a266d441d: Mounted from eboraas/apache-php
777f846aa90b: Mounted from eboraas/apache-php
1bf334fa8eab: Mounted from eboraas/apache-php
c3dd96001c2a: Mounted from eboraas/apache-php
2a6131694795: Mounted from eboraas/apache-php
22b45783e204: Mounted from eboraas/apache-php
01ce5d420f62: Mounted from eboraas/apache-php
00b3256d8d5c: Mounted from eboraas/apache-php
e306a95461ac: Layer already exists
0.2: digest: sha256:647b6c64756986701ed6e35e0de3837df48a2402422003989681b8ab22bbcb4d size: 3029
The push refers to a repository [docker.io/likid0/liquidapp]
86f91eb1cf0d: Layer already exists
6eac29325f1a: Layer already exists
06f22bc3a995: Layer already exists
d2223eb091a9: Layer already exists
876a266d441d: Layer already exists
777f846aa90b: Layer already exists
1bf334fa8eab: Layer already exists
c3dd96001c2a: Layer already exists
2a6131694795: Layer already exists
22b45783e204: Layer already exists
01ce5d420f62: Layer already exists
00b3256d8d5c: Layer already exists
e306a95461ac: Layer already exists
release: digest: sha256:647b6c64756986701ed6e35e0de3837df48a2402422003989681b8ab22bbcb4d size: 3029
Publish complete
Just to check we can pull with the new tags from github.
docker pull likid0/liquidapp:0.2 (10-31 12:14)
0.1: Pulling from likid0/liquidapp
Digest: sha256:647b6c64756986701ed6e35e0de3837df48a2402422003989681b8ab22bbcb4d
Status: Image is up to date for likid0/liquidapp:0.2
OK, so now we are able to tag and push our image in and automated fasion, on the next post we are going to create a Jenkins file, to get all this work in a pipeline.
Unix Systems:
Add new comment