For Docker deployments, CoreOS is powerful but can be complex to setup. Fig is simple, but hard to scale to multiple servers. This blog post will show you how to bridge the gap between building complex multi-container apps using Fig and deploying those applications into a production CoreOS system.

Complete Multi-Container Fig App

In last week's blog post, we covered building a 4-container app in Fig. To cut to the chase, here is the fig.yml that gets you a 4-container app:

serf:
  image: ctlc/serf
  ports:
   - 7373
   - 7946
lb:
  image: ctlc/haproxy-serf
  ports:
   - 80:80
  links:
   - serf
  environment:
   HAPROXY_PASSWORD: qa1N76pWAri9
web:
  image: ctlc/wordpress-serf
  ports:
   - 80
  environment:
   DB_PASSWORD: qa1N76pWAri9
  links:
   - serf
   - db
  volumes:
   - /local/path/to/wordpress:/app
db:
  image: ctlc/mysql-serf
  ports:
   - 3306
  volumes:
   - /mysql:/var/lib/mysql
  environment:
   MYSQL_DATABASE: wordpress
   MYSQL_ROOT_PASSWORD: qa1N76pWAri9

To show this 4-container system in action, you simply run fig up -d (which is the same command used to restart the running system).

$ fig up -d
Recreating ctlcblog_serf_1…
Recreating ctlcblog_db_1…
Recreating ctlcblog_web_1…
Creating ctlcblog_lb_1…

$ fig ps
Name               Command  State   Ports
-------------------------------------------------------------------------------
ctlcblog_serf_1    /run.sh  Up      49252->7373/tcp, 49253->7946/tcp
ctlcblog_db_1      /run.sh  Up      49254->3306/tcp
ctlcblog_web_1     /run.sh  Up      49255->80/tcp
ctlcblog_lb_1      /run.sh  Up      80->80/tcp

$ fig scale web=2
Starting ctlcblog_web_2...

$ fig ps
Name               Command  State   Ports
-------------------------------------------------------------------------------
ctlcblog_serf_1    /run.sh  Up      49252->7373/tcp, 49253->7946/tcp
ctlcblog_db_1      /run.sh  Up      49254->3306/tcp
ctlcblog_web_2     /run.sh  Up      49256->80/tcp
ctlcblog_web_1     /run.sh  Up      49255->80/tcp
ctlcblog_lb_1      /run.sh  Up      80->80/tcp

As you can see, Fig is super easy to get started with and use, but the #1 question people had was how do you scale this to multiple servers?

How Do You Create CoreOS Containers From Fig Configurations?

CoreOS is not the easiest or most approachable system in the world. Understanding how to write systemd config files and getting everything configured just right can be time consuming and frustrating.

Which is why I wrote a fig2coreos gem (github.com/centurylinklabs/fig2coreos), which converts fig.yml to CoreOS formatted systemd configuration files automatically.

$ sudo gem install fig2coreos
$ fig2coreos -t vagrant wordpress-app fig.yml coreos-dir
[SUCCESS] Try this: cd /Users/cardmagic/Sites/centurylinklabs.com/coreos-dir && vagrant up
$ cd coreos-dir && vagrant up
Bringing machine 'default' up with 'virtualbox' provider…
[default] Importing base box 'coreos'…
[default] Matching MAC address for NAT networking…
[default] Setting the name of the VM…
[default] Clearing any previously set forwarded ports…
[default] Clearing any previously set network interfaces…
[default] Preparing network interfaces based on configuration…
[default] Forwarding ports…
[default] — 22 => 2222 (adapter 1)
[default] — 80 => 8080 (adapter 1)
[default] Running 'pre-boot' VM customizations…
[default] Booting VM…
[default] Waiting for machine to boot. This may take a few minutes…
[default] Machine booted and ready!
[default] Setting hostname…
[default] Configuring and enabling network interfaces…
[default] Exporting NFS shared folders…
Preparing to edit /etc/exports. Administrator privileges will be required…
Password:
[default] Mounting NFS shared folders…
[default] Running provisioner: shell…
[default] Running: /var/folders/9j/gkydy1sn1nsd73yd2n3l68400000gn/T/vagrant-shell20140224-18153-19i4zr8
$ cd coreos-dir && vagrant ssh
Last login: Mon Feb 24 23:57:38 UTC 2014 from 10.0.2.2 on ssh
______ ____ _____
/ ____/___ ________ / __ / ___/
/ / / __ / ___/ _ / / / /__
/ /___/ /_/ / / / __/ /_/ /___/ /
____/____/_/ ___/____//____/
[email protected] ~ $

For some reason (I am not sure why yet), the default version of CoreOS shipped when you vagrant up is not the latest version of coreos, which means it does not have Fleet installed and has an old version of Docker (0.7.2). Once the vagrant coreos box stays up and running long enough to download the latest version of coreos, you can apply the update by either running sudo reboot in the coreos image, or by running vagrant reload --provision in the generated directory.

What Does fig2coreos Do, Exactly?

The fig2coreos command parses your fig.yml file and generates a bunch of systemd config files. You can see the files in the directory that is created.

$ ls coreos-dir/media/state/units/
db-discovery.1.service
lb-discovery.1.service
serf-discovery.1.service
web-discovery.1.service
db.1.service
lb.1.service
serf.1.service
web.1.service
$ cat coreos-dir/media/state/units/db.1.service
[Unit]
Description=Run db_1
After=docker.service
Requires=docker.service

[Service]
Restart=always
RestartSec=10s
ExecStartPre=/usr/bin/docker ps -a -q | xargs docker rm
ExecStart=/usr/bin/docker run -rm -name db_1 -v /mysql:/var/lib/mysql -e "MYSQL_DATABASE=wordpress" -e "MYSQL_ROOT_PASSWORD=nqhT4hT0RT6k" -p 3306 ctlc/mysql-serf
ExecStartPost=/usr/bin/docker ps -a -q | xargs docker rm
ExecStop=/usr/bin/docker kill db_1
ExecStopPost=/usr/bin/docker ps -a -q | xargs docker rm

[Install]
WantedBy=local.target

Which looks complicated, but is the CoreOS version of this part of your fig.yml file:

db:
image: ctlc/mysql-serf
ports:
– 3306
volumes:
– /mysql:/var/lib/mysql
environment:
MYSQL_DATABASE: wordpress
MYSQL_ROOT_PASSWORD: qa1N76pWAri9

In addition to the start script for MySQL, there is also an etcd auto-registry script via systemd in the db-discovery.1.service file.

$ cat coreos-dir/media/state/units/db-discovery.1.service
[Unit]
Description=Announce db_1
BindsTo=db.1.service
[Service]
ExecStart=/bin/sh -c "while true; do etcdctl set /services/db/db_1 '{ "host": "%H", "port": 3306, "version": "52c7248a14″ }' –ttl 60;sleep 45;done"
ExecStop=/usr/bin/etcdctl rm /services/db/db_1

[X-Fleet]
X-ConditionMachineOf=db.1.service

This registers the service into the etcd key/value store. In this example Fig application, we used Serf to make the Docker containers self-aware, but etcd is the standard CoreOS alternative to Serf available in all CoreOS containers.

Do you need both Serf and etcd? No. The biggest difference between the two is where the daemons are run. In etcd, the etcd daemons are run outside of the Docker containers and it is up to a system like systemd to keep the etcd key/value pairs up to date. In Serf, the serf daemons are run inside the Docker containers, so when a container dies or goes offline, the peers of that container figure it out automatically.

These are two very different philosophies for configuration and system management, but in the end result is essentially the same. Both Serf and etcd have the same ability to let Docker apps become more aware of their containers and the roles their containers play and interconnect.

So How Do You Scale Docker To Multiple Hosts?

You can (in theory) use Fig on each one of your servers, each running independent Docker daemons. However linking containers becomes more complicated and network issues make it non-trivial, even with Serf.

I am getting closer to showing how to deploy multi-host Docker in this series of blog posts, but this post won't show you exactly how just yet. Subscribe to get weekly updates and we will get there in due time.

In the meantime, so that you know that I am not lying to you, watch this 2-minute YouTube Video about CoreOS Fleet to see an example of a real multi-host CoreOS deployment. You can see the power of CoreOS and Fleet in this simple video, but it doesn't connect the dots of how to put all the components together to re-create the demo.

Conclusion

We are getting very close to showing you how to deploy production multi-host Docker apps in a cloud environment. Each week, I am getting you closer and closer to the answer by introducing slightly more complexity into the example. CoreOS and Fig are just two Docker technologies being built to manage complex applications. In the future we will show you other ways you may want to accomplish this same task with tools like Flynn and Deis.