In this series of blog posts, we have progressively built increasingly complex Docker-based applications. We started with a simple 2-container Docker app. Then we built a 3-container decentralized Docker app with Serf.

(Edit: Fig was purchased by Docker in mid-2014, and renamed Docker Compose. As such, all Fig references now apply to Docker Compose.)

Today we are building a 4-container Docker app that uses Fig, HAProxy and Serf. When you start building apps this complex with Docker, maintaining the state of your containers can become very difficult. There are multiple ways to manage this complexity, but this we will cover just one: Fig from the makers of Orchard.

2-Container (Apache, MySQL) Example

Fig is a great way to keep all the state of your complex application in one place. Let's look at a fig configuration that is equivalent to the 2-container system we built in our first blog post.

web:
  image: myuser/wordpress
  ports:
    - 80:80
  environment:
    - DB_PASSWORD: qa1N76pWAri9
db:
  image: myuser/mysql
  ports:
    - 3306
  volumes:
    - /mysql:/var/lib/mysql

And to run this fig.yml file, you put it in your current working directory and run:

$ fig up -d
Creating ctlcblog_db_1...
Creating ctlcblog_web_1...
$ fig ps
Name            Command  State  Ports  
-------------------------
ctlcblog_db_1   /run.sh  Up     49250->3306/tcp  
ctlcblog_web_1  /run.sh  Up     80->80/tcp

PROS

  • Simplest fig.yml file
  • Compared to Apache and MySQL in the same container, you can restart and manage Apache separately from MySQL
  • Simple to manage and run just 2 Docker containers

CONS

  • You have to create custom containers called myuser/wordpress and myuser/mysql which is time consuming and difficult
  • Scaling from this simple setup is hard
  • A lot of configuration is hard coded into the containers which is not ideal

3-Container (Serf, Apache, MySQL) Example

Adding Serf is a great way to get your containers to auto-discover each other. Let's look at a fig configuration that is equivalent to the 3-container system we built in our second blog post.

serf:
  image: ctlc/serf
  ports:
    - 7373
    - 7946
web:
  image: ctlc/wordpress-serf
  ports: - 80:80
  environment:
    DB_PASSWORD: qa1N76pWAri9
  links:
    - serf
db:
  image: myuser/mysql
  ports:
    - 3306
  links:
    - serf
  volumes:
    - /mysql:/var/lib/mysql

And to run this fig.yml file, you put it in your current working directory and run:

$ fig up -d
Creating ctlcblog_serf_1...
Recreating ctlcblog_db_1...
Recreating ctlcblog_web_1...
$ fig ps
Name             Command  State  Ports  
---------------------------------
ctlcblog_serf_1  /run.sh  Up     49248->7373/tcp, 49249->7946/tcp
ctlcblog_db_1    /run.sh  Up     49250->3306/tcp  
ctlcblog_web_1   /run.sh  Up     80->80/tcp

PROS

  • WordPress container stays generic because of the auto-generated configs from Serf information
  • Generic auto-generated configuration based on Serf's clustering logic which lets us leverage Serf to auto-detect MySQL database (notice that we don't link the db container to the web container, but we link both containers to the serf container which figures out how to bind them together)
  • Still simple to manage and run 3 Docker containers

CONS

  • Adding more ctlc/wordpress-serf containers doesn't load balance or scale automatically
  • You still have to create a custom MySQL containers called myuser/mysql which is time consuming and a pain
  • This setup doesn't take advantage of Serf's ability to do event hooks like adding more members or removing members from the system

NEW: 4-Container (Serf, HAProxy, Apache, MySQL) Example

Fig can automatically build the links between Docker containers and save you the time and hassle of writing docker run -link commands yourself. However Fig can not talk to the containers themselves, so if you add a new web server via Fig's scale command (e.g., fig scale web=3), you would need to restart any load balancers to take advantage of the new environmental variables which could mean downtime for your users. This is where Serf combined with Fig shines.

The most powerful part of Serf are the event hooks, and the previous examples did not show off this power. To see the power of Serf's event hooks, you can create an haproxy container.

Luckily I have created a ctlc/haproxy-serf Trusted Build Image with Serf built-in for you to use. This image has a different serf agent call than the other images. For reference, let's first look at the serf agent call in the WordPress container. You can see it on GitHub if you want to.

$ serf agent -tag role=web

Ok, simple enough.

So then what does the haproxy serf look like? You can check it out on GitHub but here it is if you don't want to click:

serf agent -tag role=lb
  -event-handler="member-join=/serf-member-join.sh"  
  -event-handler="member-leave,member-failed=/serf-member-leave.sh"

Clearly a lot more going on. Let's break it down. First let's look at serf-member-join.sh:

if [ "x${SERF_SELF_ROLE}" != "xlb" ]; then echo "Not an lb. Ignoring member join." exit 0 fi

while read line; do
    ROLE=`echo $line | awk '{print $3 }'`
    if [ "x${ROLE}" != "xweb" ]; then
        continue
    fi

    echo $line |
        awk '{ printf "    server %s %s checkn", $1, $2 }' >>/etc/haproxy/haproxy.cfg
done

service haproxy reload

This bash script will be called every time a new Serf agent is added to the network. The script looks for only the web servers and adds them to the haproxy config file. Then at the end, it gracefully restarts haproxy with no downtime. Cool, right? I won't paste the code for serf-member-leave.sh, but it un-does the above work.

Now let's add haproxy to the fig.yml.

serf:
  image: ctlc/serf
  ports:
    - 7373
    - 7946
lb:
  image: ctlc/haproxy-serf
  ports:
    - 80:80
  links:
    - serf
  environment:
    HAPROXY_PASSWORD: qa1N76pWAri9
web:
  image: ctlc/wordpress-serf
  ports:
    - 80
  environment:
    DB_PASSWORD: qa1N76pWAri9
  links:
    - serf
    - db
  volumes:
    - /local/path/to/wordpress:/app
db:
  image: ctlc/mysql
  ports:
    - 3306
  volumes:
    - /mysql:/var/lib/mysql
  environment:
    MYSQL_DATABASE: wordpress
    MYSQL_ROOT_PASSWORD: qa1N76pWAri9

Notice something interesting here, I actually replaced the DB image from ctlc/mysql-serf to ctlc/mysql and the app still works. I did this to make a point. The ctlc/mysql container doesn't have a Serf agent in it like ctlc/mysql-serf does, so in this example we use a hybrid of Serf detection for the load balancing and Docker linking for the database connection string.

The upside to using the ctlc/mysql container is that you can set the MySQL database and root password through environmental variables instead of having to build a custom local image that has a random password assigned. To show this 4-container system in action, you simply run fig up -d (which is the same command used to restart the running system).

$ fig up -d Recreating ctlc
blog_serf_1...
Recreating ctlcblog_db_1...
Recreating ctlcblog_web_1...
Creating ctlcblog_lb_1...

$ fig ps
     Name         Command              State   Ports               
-------------------------------------------------------------------------------
ctlcblog_serf_1   /run.sh              Up      49252->7373/tcp, 49253->7946/tcp
ctlcblog_db_1     /usr/local/bin/run   Up      49254->3306/tcp                  
ctlcblog_web_1    /run.sh              Up      49255->80/tcp                    
ctlcblog_lb_1     /run.sh              Up      80->80/tcp                       

$ fig scale web=2
Starting ctlcblog_web_2...

$ fig ps
     Name         Command              State   Ports               
-------------------------------------------------------------------------------
ctlcblog_serf_1   /run.sh              Up      49252->7373/tcp, 49253->7946/tcp
ctlcblog_db_1     /usr/local/bin/run   Up      49254->3306/tcp                  
ctlcblog_web_2    /run.sh              Up      49256->80/tcp                    
ctlcblog_web_1    /run.sh              Up      49255->80/tcp                    
ctlcblog_lb_1     /run.sh              Up      80->80/tcp

PROS

  • You no longer need to create custom containers to build your app, all the containers are completely self-configuring and self-aware of all the other containers
  • Easy to scale up and down the web tier by running fig scale web=5 and the load balancer will auto-detect new web nodes
  • Simple to manage through fig

CONS

  • More complex fig.yml file
  • Doesn't yet work on multiple hosts (we will talk about this in future blog posts)
  • Difficult to setup manually if you don't like to use fig

Conclusion

Sticking HAProxy, Apache and MySQL in a single Docker container might be the easiest way to get a site up and running quickly, but it doesn't scale. Getting a Docker-based system to scale is not rocket science, but takes some careful thought and attention. The projects and tools being built around Docker today can make your life a lot easier if you know how to use them correctly.

In the coming weeks we will look at multi-host Docker solutions and alternatives to Fig and Serf like CoreOS.