You are here

Continuous Delivery Workflow example with Docker Docker-Compose and Ansible

Hi

We are going to start the journey on creating the Continuous deployment Workflow pipeline for our prize winner php app.

We are taking on from the previous post, check it out to get up to date, with what we have done:

http://hpuxtips.es/?q=content/docker-using-docker-compose-build-lamp-dev...

Git repo with all the files we work with in these posts:
https://github.com/likid0/CDdockerAnsible

We are using such a uselless and simple PHP app , so we can concetrate on creating the workflow and don't waste any time with the application itself.

We are going to divide our workflow in 3 stages, dev ,release and deploy, we are missing a build stage but the app is so simple that we don't have anything to compile.

In the first stage of our pipeline Our Goal is that once we have finished coding and commited the changes to our app, we wan't to be able run some basic integration tests, just to test everything is working ok in our app internally, And that our new updates to the app haven't broken anything.

So first of all, I'm going to modify our web Docker image to include a script as the ENTRYPOINT, this script gives us 2 options to start the webserver that runs the app, or run our integration tests to check the app functionality, and also check if the conection to the mysql database is working.

[liquid@liquid-ibm:liquid-app/scripts]$ pwd                                                                                                                                              (10-24 22:23)
/home/liquid/lamp-docker/liquid-app/scripts
[liquid@liquid-ibm:liquid-app/scripts]$ cat run-script.sh                                                                                                                                (10-24 22:23)
#!/bin/bash
case  $1  in
start)
#It's important to use the exec shell built in command to replace the shell with the apache web server proc.
#So apache proc takes pid 1 and when we do a stop of the container we get our proc stopped with a gracious kill
exec /usr/sbin/apache2ctl -D FOREGROUND
;;
test)
#we use the php cli to run the app and check it returns valid entries from the Database.
if [ $(/usr/bin/php /var/www/html/index.php | grep IP | wc -l ) -gt 0 ] ; then
   echo "PHP app Test Passed OK"
else
   echo "PHP app Test Failed" 
fi
;;
*)
echo "please use $@ start|test"
;;
esac

How do we add this script to our docker file:

[liquid@liquid-ibm:docker/dev]$ cat Dockerfile                                                                                                                                           (10-24 22:34)
FROM eboraas/apache-php

MAINTAINER Daniel Alexander Parkes
LABEL version="0.2"

RUN  echo 'export DBNAME=${DBNAME}\n\
export DBUSER=${DBUSER}\n\
export DBPASS=${DBPASS}\n\
export DBHOST=${DBHOST}\n' >> /etc/apache2/envvars

RUN sed -i -e 's/DirectoryIndex.*$/DirectoryIndex index.php/g' /etc/apache2/mods-available/dir.conf && rm /etc/apache2/sites-enabled/default-ssl.conf

#We add the script to our image and make it executable
COPY scripts/run-script.sh /usr/local/bin/run-script.sh
RUN chmod +x /usr/local/bin/run-script.sh

EXPOSE 80 

#We use our script as the entrypoint, so we have the option to run in start or test mode
ENTRYPOINT ["run-script.sh"]
#By default we run in start mode
CMD ["start"]

lets rebuild our image and start our env with the new image:

With -v option we remove all related volumes:

[liquid@liquid-ibm:docker/dev]$ docker-compose down -v                                                                                                                                   (10-24 22:34)
Stopping dev_lb_1 ... done
Stopping dev_web_1 ... done
Stopping dev_db_1 ... done
Removing dev_web_run_1 ... done
Removing dev_lb_1 ... done
Removing dev_web_1 ... done
Removing dev_db_1 ... done
Removing network dev_default
Removing volume dev_webroot

[liquid@liquid-ibm:docker/dev]$ docker-compose up -d --build                                                                                                                             (10-24 22:36)
Creating network "dev_default" with the default driver
Creating volume "dev_webroot" with local driver
Building web
Step 1 : FROM eboraas/apache-php
 ---> 80012ded4d94
Step 2 : MAINTAINER Daniel Alexander Parkes
 ---> Using cache
 ---> 862823a41a02
Step 3 : LABEL version "0.2"
 ---> Using cache
 ---> 036001e286ae
Step 4 : RUN echo 'export DBNAME=${DBNAME}\nexport DBUSER=${DBUSER}\nexport DBPASS=${DBPASS}\nexport DBHOST=${DBHOST}\n' >> /etc/apache2/envvars
 ---> Using cache
 ---> 233d9ddb7b37
Step 5 : RUN sed -i -e 's/DirectoryIndex.*$/DirectoryIndex index.php/g' /etc/apache2/mods-available/dir.conf && rm /etc/apache2/sites-enabled/default-ssl.conf
 ---> Using cache
 ---> bec531a949ab
Step 6 : COPY scripts/run-script.sh /usr/local/bin/run-script.sh
 ---> Using cache
 ---> 62a67aad5227
Step 7 : RUN chmod +x /usr/local/bin/run-script.sh
 ---> Using cache
 ---> 6d67aba08467
Step 8 : EXPOSE 80
 ---> Using cache
 ---> 87a0d26e869b
Step 9 : ENTRYPOINT run-script.sh
 ---> Using cache
 ---> f9a6c5a14d10
Step 10 : CMD start
 ---> Using cache
 ---> 4eb3d7dd0663
Successfully built 4eb3d7dd0663
Creating dev_db_1
Creating dev_web_1
Creating dev_lb_1

[liquid@liquid-ibm:docker/dev]$ docker-compose ps                                                                                                                                        (10-24 22:37)
  Name                 Command               State                   Ports                 
------------------------------------------------------------------------------------------
dev_db_1    docker-entrypoint.sh mysqld      Up      3306/tcp                              
dev_lb_1    /sbin/tini -- dockercloud- ...   Up      1936/tcp, 443/tcp, 0.0.0.0:80->80/tcp 
dev_web_1   run-script.sh start              Up      443/tcp, 80/tcp       

We can now invoke our integration tests using the test CMD, here we use the docker-compose run command,
normall used to Run a one-off command on a service, as in this case to run the tests:

[liquid@liquid-ibm:docker/dev]$ docker-compose run web test                                                                                                                              (10-24 22:39)
PHP Warning:  mysqli::mysqli(): (HY000/2003): Can't connect to MySQL server on 'db' (111) in /var/www/html/index.php on line 20
PHP app Test Failed

As you can see our tests have failed, we can't connect to the database, we check that all db parameters are ok, but when we run a telnet to
port 3306 it doesn't respond, so the problem is we are running the tests before our DB has initialized.

To verify that is the problem, we run the command a minute after and the tests pass ok:

[liquid@liquid-ibm:docker/dev]$ docker-compose run web test                                                                                                                              (10-24 22:41)
PHP app Test Passed OK

As of today there is no way in docker-compose that you can controll that the application the container runs has finished it's initialization.

So what we are going to use is a container agent that waits until the DBs is running
we are going to use ansible and it's module wait_for

We are going to create a dir file four our ansible image:

[liquid@liquid-ibm:lamp-docker/ansible]$ pwd                                                                                                                                             (10-24 23:36)
/home/liquid/lamp-docker/ansible
[liquid@liquid-ibm:lamp-docker/ansible]$ tree                                                                                                                                            (10-24 23:36)
.
├── Dockerfile
└── probe-service.yml

0 directories, 2 files

[liquid@liquid-ibm:lamp-docker/ansible]$ cat Dockerfile                                                                                                                                  (10-24 23:36)
from ubuntu:trusty

# Install Ansible
RUN apt-get update -qy && \
    apt-get install -qy software-properties-common && \
    apt-add-repository -y ppa:ansible/ansible && \
    apt-get update -qy && \
    apt-get install -qy ansible

# Copy baked in playbooks
COPY probe-service.yml /ansible/

# Add volume for Ansible playbooks
VOLUME /ansible
WORKDIR /ansible

# Entrypoint
ENTRYPOINT ["ansible-playbook"]
CMD ["probe-service.yml"]  

And our playbook:

---
- name: Probe a service via tcp port
  hosts: localhost
  connection: local  ---> important when running on localhost
  gather_facts: no
  tasks:
  - name: Set facts
    set_fact:       ----> here we get our envs from the shell variables passed on with docker
      probe_host: "{{ lookup('env','PROBE_HOST') }}"
      probe_port: "{{ lookup('env','PROBE_PORT') }}"
      probe_delay: "{{ lookup('env','PROBE_DELAY') | default(0, true) }}"
      probe_timeout: "{{ lookup('env','PROBE_TIMEOUT') | default (80, true) }}"
  - name: Waiting for DB 
    local_action: >
      wait_for host={{ probe_host }}  ---> the wait_for module, it wont exit until the host responds on that port
      port={{ probe_port }}           ---> or the timeout is reached
      delay={{ probe_delay }}
      timeout={{ probe_timeout }}

[liquid@liquid-ibm:lamp-docker/ansible]$ docker build -t ansible-agent .                                                                                                                 (10-24 23:40)
Sending build context to Docker daemon 3.584 kB 
Step 1 : FROM ubuntu:trusty
 ---> 1e0c3dd64ccd
Step 2 : RUN apt-get update -qy &&     apt-get install -qy software-properties-common &&     apt-add-repository -y ppa:ansible/ansible &&     apt-get update -qy &&     apt-get install -qy ansible
 ---> Using cache
 ---> 173d5e6ddf4a
Step 3 : COPY probe-service.yml /ansible/
 ---> Using cache
 ---> cb8a2891f077
Step 4 : VOLUME /ansible
 ---> Using cache
 ---> d6727ba4dec4
Step 5 : WORKDIR /ansible
 ---> Using cache
 ---> f918c3c6fdc4
Step 6 : ENTRYPOINT ansible-playbook
 ---> Running in 3cef86652665
 ---> d6d5f323ef3c
Removing intermediate container 3cef86652665
Step 7 : CMD probe-service.yml
 ---> Running in 7ada3c83e5cb
 ---> a1d86d4e13ac
Removing intermediate container 7ada3c83e5cb
Successfully built a1d86d4e13ac


[liquid@liquid-ibm:lamp-docker/ansible]$ docker images | grep -i ansible                                                                                                                 (10-24 23:40)
ansible-agent                            latest              a1d86d4e13ac        20 seconds ago      304.6 MB

I will add the service to our docker-compose.yml file:

[liquid@liquid-ibm:docker/dev]$ pwd                                                                                                                                                      (10-24 23:43)
/home/liquid/lamp-docker/liquid-app/docker/dev
[liquid@liquid-ibm:docker/dev]$ tree                                                                                                                                                     (10-24 23:43)
.
├── docker-compose.yml
└── Dockerfile

0 directories, 2 files

On the compose file I have removed all the load balancer references, for integration tests we are only testing the app funcionality so it's not necesary

[liquid@liquid-ibm:docker/dev]$ cat docker-compose.yml                                                                                                                                   (10-24 23:43)
version: '2'

volumes:
  webroot:
    driver: local
services:
  web:
    build: 
      context: ../../
      dockerfile: docker/dev/Dockerfile
    volumes:
      - ../../src:/var/www/html
    links:
      - db
    environment:
      DBNAME: liquidapp
      DBUSER: liquiduser
      DBPASS: liquiduser
      DBHOST: db
# we have also added the command statement, so when we run the up command the app tests will be run by default
    command: ["test"]


  db: 
    image: mysql:5.6
    hostname: db
    expose:
      - "3306"
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_USER: liquiduser
      MYSQL_PASSWORD: liquiduser
      MYSQL_DATABASE: liquidapp
#  lb:
#    image: dockercloud/haproxy
#    links:
#      - web
#    volumes:
#      - /var/run/docker.sock:/var/run/docker.sock
#    ports:
#      - 80:80

#here we add our agent service linking it to the db service and passing the database host and port as envars 
  agent:
    image: ansible-agent
    links:
      - db
    environment:
      PROBE_HOST: "db"
      PROBE_PORT: "3306"

Ok, lets test the agent:

[liquid@liquid-ibm:docker/dev]$ docker-compose up  agent                                                                                                                                 (10-24 23:35)
Creating network "dev_default" with the default driver
Creating volume "dev_webroot" with local driver
Creating dev_db_1
Creating dev_agent_1
Attaching to dev_agent_1
agent_1  |  [WARNING]: provided hosts list is empty, only localhost is available
agent_1  | 
agent_1  | PLAY [Probe a service via tcp port] ********************************************
agent_1  | 
agent_1  | TASK [Set facts] ***************************************************************
agent_1  | ok: [localhost]
agent_1  | 
agent_1  | TASK [Waiting for DB] **********************************************************
agent_1  | ok: [localhost -> localhost]
agent_1  | 
agent_1  | PLAY RECAP *********************************************************************
agent_1  | localhost                  : ok=2    changed=0    unreachable=0    failed=0   
agent_1  | 
dev_agent_1 exited with code 0

Ok, so now we now our DB initialized, before we run our tests, lets try the full workflow:

[liquid@liquid-ibm:docker/dev]$ docker-compose down -v                                                                                                                                   (10-24 23:43)
Stopping dev_db_1 ... done
Removing dev_agent_1 ... done
Removing dev_db_1 ... done
Removing network dev_default
Removing volume dev_webroot

[liquid@liquid-ibm:docker/dev]$ docker-compose up agent                                                                                                                                  (10-24 23:50)
Creating network "dev_default" with the default driver
Creating volume "dev_webroot" with local driver
Creating dev_db_1
Creating dev_agent_1
Attaching to dev_agent_1
agent_1  |  [WARNING]: provided hosts list is empty, only localhost is available
agent_1  | 
agent_1  | PLAY [Probe a service via tcp port] ********************************************
agent_1  | 
agent_1  | TASK [Set facts] ***************************************************************
agent_1  | ok: [localhost]
agent_1  | 
agent_1  | TASK [Waiting for DB] **********************************************************
agent_1  | ok: [localhost -> localhost]
agent_1  | 
agent_1  | PLAY RECAP *********************************************************************
agent_1  | localhost                  : ok=2    changed=0    unreachable=0    failed=0   
agent_1  | 
dev_agent_1 exited with code 0
[liquid@liquid-ibm:docker/dev]$ docker-compose up web                                                                                                                                    (10-24 23:51)
dev_db_1 is up-to-date
Recreating dev_web_1
Attaching to dev_web_1
web_1    | PHP app Test Passed OK   --------> ok tests passed.
dev_web_1 exited with code 0

So this is our dev enviroment test workflow finished, we are going to add it to a make file from where we are going to control our build stages:

[liquid@liquid-ibm:lamp-docker/liquid-app]$ pwd                                                                                                                                          (10-25 00:17)
/home/liquid/lamp-docker/liquid-app
[liquid@liquid-ibm:lamp-docker/liquid-app]$ tree                                                                                                                                         (10-25 00:17)
.
├── docker
│   ├── dev
│   │   ├── docker-compose.yml
│   │   └── Dockerfile
│   └── release
├── Makefile
├── scripts
│   └── run-script.sh
└── src
    └── index.php

5 directories, 5 files
[liquid@liquid-ibm:lamp-docker/liquid-app]$ cat Makefile                                                                                                                                 (10-25 00:17)
# Here we create our vars for the docker-compose project and our DEV file
DEV_PRO := liquidappdev
DEV_FILE := docker/dev/docker-compose.yml


.PHONY: test build release clean

test:
	@ echo "Pulling latest images..."
# First we pull the latest images from docker hub in this case we only have db
	@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) pull db 
# Then we build our web container using the --pull option to download the latest image
# that the Docker file has referenced in the From directive
	@ echo "Building images..."
	@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) build --pull web
# We run the agent with the --rm option so it deletes the container after the run
	@ echo "Ensuring database is ready..."
	@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) run --rm agent
# We finally run our tests
	@ echo "Running tests..."
	@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) up web

clean:
	@ echo "Destroying dev env"
# here we clean up all the objects in the enviroment including those anoying dangling images.
	@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) kill
	@ docker-compose -p $(DEV_PRO) -f $(DEV_FILE) rm --all -f -v
	@ echo "Removing dangling objects"
	@ docker images -q -f dangling=true -f label=application=$(REPO_NAME) | xargs -I ARGS docker rmi -f ARGS
	@ echo "Clean complete"

ok let's try it out:

[liquid@liquid-ibm:lamp-docker/liquid-app]$ make test                                                                                                                                    (10-25 00:15)
echo "Pulling latest images..."
Pulling latest images...
Pulling db (mysql:5.6)...
5.6: Pulling from library/mysql
Digest: sha256:b714e9bd05f38877832a976813dec03650c3f4a4b09cd23652a85efd41ae6cd8
Status: Image is up to date for mysql:5.6
echo "Building images..."
Building images...
Building web
Step 1 : FROM eboraas/apache-php
latest: Pulling from eboraas/apache-php

ec029f31699b: Pull complete
7063201e491e: Pull complete
be5b48161e1a: Pull complete
88c7f9f9ea24: Pull complete
98c0d74d2f5c: Pull complete
c75b4803d2ba: Pull complete
9ae30326fdf3: Pull complete
59eb494813db: Pull complete
93a42cad86c2: Pull complete
Digest: sha256:3819c90f2706c24904728f0a23312f21891da6b832d5984efaec12e72deb2e21
Status: Downloaded newer image for eboraas/apache-php:latest
 ---> 5c50c1deba2c
Step 2 : MAINTAINER Daniel Alexander Parkes
 ---> Running in 7814a8063242
 ---> 345535acf24a
Removing intermediate container 7814a8063242
Step 3 : LABEL version "0.2"
 ---> Running in cbab24b0d4b8
 ---> 13d05c8df803
Removing intermediate container cbab24b0d4b8
Step 4 : RUN echo 'export DBNAME=${DBNAME}\nexport DBUSER=${DBUSER}\nexport DBPASS=${DBPASS}\nexport DBHOST=${DBHOST}\n' >> /etc/apache2/envvars
 ---> Running in 86b28b4e0f6a
 ---> b1062cdcc5f4
Removing intermediate container 86b28b4e0f6a
Step 5 : RUN sed -i -e 's/DirectoryIndex.*$/DirectoryIndex index.php/g' /etc/apache2/mods-available/dir.conf && rm /etc/apache2/sites-enabled/default-ssl.conf
 ---> Running in 5c530337b6a3
 ---> 6286a1fb7da5
Removing intermediate container 5c530337b6a3
Step 6 : COPY scripts/run-script.sh /usr/local/bin/run-script.sh
 ---> 4b0a009eda5d
Removing intermediate container a469c6b52e3a
Step 7 : RUN chmod +x /usr/local/bin/run-script.sh
 ---> Running in 52f3eea7e081
 ---> 39580e0be540
Removing intermediate container 52f3eea7e081
Step 8 : EXPOSE 80
 ---> Running in beceab9c388d
 ---> 22230f68be3b
Removing intermediate container beceab9c388d
Step 9 : ENTRYPOINT run-script.sh
 ---> Running in bcab2b82afc0
 ---> 030858e32b2a
Removing intermediate container bcab2b82afc0
Step 10 : CMD start
 ---> Running in d53905722183
 ---> 275178a889da
Removing intermediate container d53905722183
Successfully built 275178a889da
echo "Ensuring database is ready..."
Ensuring database is ready...
Creating network "liquidappdev_default" with the default driver
Creating volume "liquidappdev_webroot" with local driver
Creating liquidappdev_db_1
 [WARNING]: provided hosts list is empty, only localhost is available


PLAY [Probe a service via tcp port] ********************************************

TASK [Set facts] ***************************************************************
ok: [localhost]

TASK [Waiting for DB] **********************************************************
ok: [localhost -> localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0   

echo "Running tests..."
Running tests...
liquidappdev_db_1 is up-to-date
Creating liquidappdev_web_1
Attaching to liquidappdev_web_1
web_1    | PHP app Test Passed OK    -----------> tests run fine
liquidappdev_web_1 exited with code 0

Lets try to clean up shop:

[liquid@liquid-ibm:lamp-docker/liquid-app]$ make clean                                                                                                                                   (10-25 00:16)
echo "Destroying dev env"
Destroying dev env
Killing liquidappdev_db_1 ... done
This will be the default behavior in the next version of Compose.

Going to remove liquidappdev_web_1, liquidappdev_db_1
Removing liquidappdev_web_1 ... done
Removing liquidappdev_db_1 ... done
echo "Removing dangling objects"
Removing dangling objects
echo "Clean complete"
Clean complete

Ok so here we have our firs step in the CD workflow pipeline, on the next post we will work on the release phase.

Regards

Unix Systems: 

Add new comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.
Error | HP-UX Tips & Tricks Site

Error

Error message

  • Warning: Cannot modify header information - headers already sent by (output started at /homepages/37/d228974590/htdocs/includes/common.inc:2567) in drupal_send_headers() (line 1207 of /homepages/37/d228974590/htdocs/includes/bootstrap.inc).
  • PDOException: SQLSTATE[42000]: Syntax error or access violation: 1142 INSERT command denied to user 'dbo229817041'@'217.160.155.192' for table 'watchdog': INSERT INTO {watchdog} (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9); Array ( [:db_insert_placeholder_0] => 0 [:db_insert_placeholder_1] => cron [:db_insert_placeholder_2] => Attempting to re-run cron while it is already running. [:db_insert_placeholder_3] => a:0:{} [:db_insert_placeholder_4] => 4 [:db_insert_placeholder_5] => [:db_insert_placeholder_6] => http://www.hpuxtips.es/?q=content/continuous-delivery-workflow-example-docker-docker-compose-and-ansible [:db_insert_placeholder_7] => [:db_insert_placeholder_8] => 54.90.207.75 [:db_insert_placeholder_9] => 1512951772 ) in dblog_watchdog() (line 157 of /homepages/37/d228974590/htdocs/modules/dblog/dblog.module).
The website encountered an unexpected error. Please try again later.