Docker is great for developing and deploying distributed applications, but many developers still don't understand how to effectively use containers yet.

This confusion exists in part because Docker's usefulness during development isn't as a replacement for a virtual machine. During development, having containers running production-ready images within your development environment is invaluable. However, if you're not building your piece of the distributed application in-container, how can you be confident the entire system will work in production just as it does in development?

When we set out to build Panamax, we knew we wanted it to be a distributed application with all the services running as micro-services inside containers. We also knew that we didn't want to rebuild images every time we changed code. This meant that we needed to be developing the various components of the system in containers running images that would later have as few additional image layers as possible. Ideally, we wanted to just add the code for the component to a base image and then rely on runtime configuration.

Developing in-containers isn't all that straightforward. You want your images to be lightweight, so you can't load them up with packages that are needed during development. Additionally, for many people, Linux isn't their primary development environment, so dealing with containers is best done in a VM. In the case of Panamax, we were reliant upon CoreOS as the Docker host VM and it's not really a development system. We had the additional requirements of needing to access the Docker Remote API without exposing the Remote API via TCP and to send jobs to Fleet, both from within a container. So how did we do it?

Synced Folders

Most of us in CenturyLink Labs use Macs as our primary development environment, so the first thing we needed to do was get code from the Mac into the VM. We had already chosen Vagrant and VirtualBox for running CoreOS, so it simply required setting up a synced folder in the Vagrantfile already used to run the VM.

config.vm.synced_folder "/Users/dharmamike/panamax/panamax-api", "/home/core/panamax-api",
id: "api",
:nfs => true,
:mount_options => ['nolock,vers=3,udp']

In this way, we can point the tools we are familiar with at the local source directory while relying on NFS to seamlessly keep the code we change locally up-to-date inside CoreOS. Vagrant also provides other options for synced folders if you don't want to use NFS.

Bind Mounts

The next step was to get the code on the CoreOS VM file system into a Docker container. You might think the simplest way to do this is to add a Dockerfile to the source code root (i.e. the directory being synced) with the image build instructions, but this just lets you build an image with the source code 'frozen' to the version in place when the image is built. Every time you change the code, you have to build the image again. This is probably where most developers get discouraged and decide to ignore containers until it's time to deploy.

We realized we could create a base image that had everything needed to run our application except the source code. Then we could run the image in the host VM in interactive mode and bind mount the source code directory as a volume, thereby injecting the code into the container and keeping it up to date as the code changed on the file system. Given the synced folder setup in the previous code snippet, this looks something like:

docker run -it
-v /home/core/panamax-api:/var/app/panamax-api centurylink/panamax-ruby-base:latest /bin/bash

The -v flag sets up the bind mount, and the -it flags together run the container interactively. Passing the /bin/bash command brings up the bash shell in the container in a terminal, and we can navigate to /var/app/panamax-api and run the Rails application that is the Panamax API.

Folder Sync

Ports

These services aren't useful if we can't interact with them, so in the same way we use a combination of synced folders and bind mounts to get code into the container, we use a series of port forwards to get messages into and out of an application.

When running the CoreOS Vagrant box, we can forward a port from the VM to the local development environment by adding another line to the Vagrantfile:

config.vm.network :forwarded_port, guest: 8888, host: 8888

This makes anything exposed on port 8888 of the guest VM accessible through port 8888 on the host development environment.

Subsequently, we can forward any port on our container to port 8888 on the guest VM (now acting as the host to the container) with the Docker -p flag when running the container. So the docker run code snippet above would change to:

docker run -it
-p 8888:3000
-v /home/core/panamax-api:/var/app/panamax-api centurylink/panamax-ruby-base:latest /bin/bash

In the case of Panamax, the Rails app will run on port 3000, so it is this port we map to port 8888 on the VM.

Port Mapping

What it feels like

Through this Inception-like configuration of local dev to VM to container with ports forwarded from the container to the VM to the local dev env, one can achieve a fairly nice development flow. You write and unit test code as you normally would using tools and the OS with which you are familiar. You might switch to a shell to start/stop/build the application from source, except now it's in the container's shell. And you validate the service interface locally via the ports that pass down into and up from the application running in-container. When everything's working as you expect, you make use of a Dockerfile to build a new image FROM the base image you've been using all along, ADD your code and execute a CMD to start/stop/build when the image is run. Lastly, you use Panamax to compose the entire distributed application and watch the magic happen.