One of Docker’s greatest benefits is its ability to reduce complex software and application installations into standard, reusable images. It doesn’t get much more complex than deploying the open source Platform-as-a-Service, Cloud Foundry v2. The CF-Mini image allows developers to deploy a functional Cloud Foundry v2 PaaS in 2-steps: Pull & Run with Docker.

Note: Most of this blog entry focuses on the technical reasons for, and challenges around, bringing the CF-Mini Docker image to fruition. If this doesn't interest you and you just want to try it out, then click on the CF-Mini link above, read the documentation, and start playing around. Due to the size of the Cloud Foundry installation, a minimum of 6GB free memory is required. 16GB of total memory is recommended.

Why build this CF-Mini Docker image?

  • Cloud Foundry aims to simplify application deployments. However, the initial task of installing the PaaS stack itself can be cumbersome.
  • No "easy" local dev environment option exists for CF anymore. The "Micro CF" pre-configured VM image available with CFv1 was appealing because it just worked out of the box. With CFv2, you'll need to be a bit of a BOSH and CF expert to get a working CF install.
  • Local development options are important for folks who want to test their code or connect to internal data services before deploying to a Cloud Foundry-based PaaS.
  • Community releases are full of bugs. This CF-Mini Docker image aims to address significant challenges during installation in an attempt to offer more stable versions of chosen releases.
  • Most of all, I wanted to test Docker's limits and its promise. I was curious how far I could take a Docker image.

gif not loading

Challenges when building CF-Mini

There were many issues to work through on this project, but the following were probably the most significant & interesting to overcome. They also provided a lot in the way of learning Docker's capabilities and limitations.

Many Moving Parts

Long Docker builds, and failures 45 minutes into builds, made early development difficult. It didn't take long to realize that multiple Dockerfiles would solve many of my problems. As such, the following three images came to be:

  • cf-mini-base (latest & v2xx) -> FROM ubuntu:12.04.5 w/ Ruby 1.9.3-p484 installed
  • cf-mini-release (latest & v2xx) -> FROM cf-mini-base:v2xx w/ Cloud Foundry v2xx installed
  • cf-mini (latest & v2xx) -> FROM cf-mini-release:v2xx w/ CF cli installed & final runtime configs/cmds

Here are some of the realized benefits of segmenting this way:

  • Cloud Foundry is a large, and complex, software stack with 12+ components. Separate Dockerfiles allowed me to stabilize installation segments and focus on issues / improvements more directly.
  • The Ruby version for the BOSH cli will often stay the same for many releases, which allows for a stable base to work from.
  • Allows for very fast builds of the final cf-mini image as I tweak its runtime components.

Very Large Images

A Docker image of 13GB or larger was what I was facing after the initial completed builds. This was problematic for a number of reasons, the biggest of which was it broke Docker Registry's Automated Build every time. I ultimately figured out how to fool the CF install into thinking some unnecessary installation packages still existed by truncating them. The stack wouldn't run if these packages were removed entirely.

find ~/cf_nise_installer/cf-release/.final_builds/packages/*/ -type f -iname "*.tgz" -print0 | xargs -0 -I {} truncate {} --size 0 proved very helpful to me as a new Docker developer trying to shrink my image.

Wildcard DNS Resolution

The "cf-mini.example" domain doesn't exist, and it was chosen for this very reason. So how to make the CF installation believe it is real and resolvable without losing our Docker host's working nameservers? Dnsmasq and some scripting against resolv.conf during runtime did the trick within the container. Step-by-step client-side dnsmasq instructions for Ubuntu and Mac OS can be found on the cf-mini information page. One can't connect to CF from outside the CF-Mini Docker container without wildcard DNS resolution of this fake subdomain.

Devicemapper Storage w/ Udev Sync

After testing this image with aufs, btrfs, and overlayfs I settled on devicemapper for my storage solution. I'd still like to move to overlayfs one day when Docker fully supports it. Out of the box devicemapper worked well initially until the project's large images began to cause corruption over time. The solution was to enable "Udev Sync". Not only that, but devicemapper assigns a default Docker container filesystem size of 10GB. My image consumes most of that after startup leaving little left for application deployments to CF. This needed to be increased as well. The final solution is documented here and, once completed, results in the following output on one's Docker server:

# Server process looks like this:
$ ps -ef |grep -i docker
docker -d -s devicemapper --storage-opt dm.basesize=30G

# Docker info returns these critical components:
$ docker info
Storage Driver: devicemapper
Udev Sync Supported: true

Container IP Changes at Runtime

Every time a container is either created or started, a new internal container-side IP address is assigned. Cloud Foundry, especially in a Docker container, is heavily reliant on that IP for communication between core stack components (e.g. cloud controller, dea, health manager, etcd, router). In order to overcome this, I had to force cf-mini to reconfigure its CF install with the current IP address at container runtime.


BOSH is core to the installation of Cloud Foundry v2. BOSH was designed, however, to leverage IaaS offerings directly in order to build, scale, and manage the CF stack. BOSH also, typically, requires its own Director server. Luckily, Nise BOSH (a lightweight bosh emulator) & CFv2 Installer offered me an alternate install option without having to abandon BOSH entirely.

CF Warden Requirements

This proved to be the most difficult issue to overcome. For those not familiar with Cloud Foundry, warden is essentially CF's version of Docker... a container technology for isolating application & service deployments. So the challenge was to nest a different type of container within a Docker container. The following were some of the toughest of warden's requirements to solve:

  • The ability to execute mount commands -> Warden requires extended Docker container privileges. Adding "--privileged" to the Docker run command accomplishes this.

    docker run --privileged
  • CGroups -> Now that the root user can mount, cgroups have to be reapplied in an alternate location (/tmp/warden/cgroup/) when starting the container. This is accomplished with the following scripted runtime commands:

    mount -tcgroup -operf_event cgroup:perf_event /tmp/warden/cgroup/perf_event
    mount -tcgroup -omemory cgroup:memory /tmp/warden/cgroup/memory
    mount -tcgroup -oblkio cgroup:blkio /tmp/warden/cgroup/blkio
    mount -tcgroup -ohugetlb cgroup:hugetlb /tmp/warden/cgroup/hugetlb
    mount -tcgroup -onet_cls,net_prio cgroup:net_prio /tmp/warden/cgroup/net_prio
    mount -tcgroup -onet_cls,net_prio cgroup:net_cls /tmp/warden/cgroup/net_cls
    mount -tcgroup -ocpu,cpuacct cgroup:cpu /tmp/warden/cgroup/cpu
    mount -tcgroup -ocpu,cpuacct cgroup:cpuacct /tmp/warden/cgroup/cpuacct
    mount -tcgroup -ocpuset cgroup:cpuset /tmp/warden/cgroup/cpuset
    mount -tcgroup -odevices cgroup:devices /tmp/warden/cgroup/devices
    mount -tcgroup -ofreezer cgroup:perf_event /tmp/warden/cgroup/freezer
  • Modprobe & Warden storage -> When warden creates its containers, it first runs modprobe against Docker's host OS. In order to provide both the Docker and nested warden containers with the necessary libraries for this function, we have to mount a shared read-only volume at container runtime. This is accomplished by passing "-v /lib/modules:/lib/modules:ro" to the Docker run command.

    $ docker run --privileged -v /lib/modules:/lib/modules:ro
    # The full run command for cf-mini is:
    $ docker run --privileged -v /lib/modules:/lib/modules:ro -p 80:80 -p 443:443 -p 4443:4443 -tdi tchughesiv/cf-mini

Final Thoughts

The project is very young, but I think it's a solid start to getting Cloud Foundry up and running without spending hours configuring BOSH and CF. If nothing else, this is a great demonstration of Docker's promise and has opened my eyes to the incredible potential of this emerging technology.

Don't have a CenturyLink account? No problem. Just head over to our website and activate an account.