Docker's underlying filesystem support options have expanded (AUFS, BRTFS, etc.), but what does that mean? How can you take advantage of different Docker filesystem options?

One of the ways Docker makes containerization so easy is by managing an overlay-style filesystem, allowing containers and images to incrementally change the filesystem layout of the image without requiring large copies of multiple images kicking around. With one exception, this is a copy-on-write approach: parent layers are held read-only, and changes are reflected in the working layer.

A similar analogy would be version control systems like SVN, that store versions between revisions in the form of the differences. Docker has always supported AUFS as its preferred storage driver, but now they offer multiple backends.

They differ from one another in terms of longevity (AUFS), security (or lack thereof -- BRTFS does not support SELinux yet), and availability.

AUFS is not in the mainline Linux kernel

While AUFS has been in Ubuntu kernels for quite some time, it is not in the mainline -- that is, upstream -- Linux kernel... Nor is it likely to be anytime soon.

If you're like me, then you need a more recent kernel than the stock Ubuntu kernel in order to leverage stable firmware for your laptop's wireless device. Additionally, AUFS cannot be built as a dkms package, meaning the entire kernel must be rebuilt to support it.

Faced with a choice between sticking with native AUFS support and horrific wireless stability, building a mainline kernel with AUFS support, and using a pre-built Ubuntu mainline kernel with the proper support but switching my Docker storage backend to devicemapper, well... That's an easy choice. Lossy wifi connectivity is enough to make anyone scream, and rebuilding kernels for little profit ceased to be fun years ago.

For the purposes of this post, we're using a Ubuntu trusty (14.04 LTS) environment, with upstream Docker installed.

You can follow along at home, intrepid reader, by firing up a Vagrant sandbox of the ubuntu/trusty64 image; that's what the exercises below will be based on.

How to Change Docker Storage Drivers

We're going to work from the simple case here, that you can start from scratch. When you change the storage driver you use with docker, you lose access to your older images and containers (for hopefully obvious reasons).

For the more complex case, you can use Docker's save and load commands (and export, import) to handle creating tarballs of your containers and images. These tarballs contain the right information to be able to recreate everything across backend storage. This article gives a good overview, but remember, like anything else the process can be made far easier with a little careful scripting:

$ for img in `docker images | awk '$1 != "REPOSITORY" { print $1 }' | sort | uniq` ; do echo $img ; docker save $img | gzip - > $img.tar.gz ; done


Docker Device-Mapping Technology

The device-mapper backend leverages some pretty cool new conflations between the Linux Volume Manager (lvm2) and the device-mapper subsystem itself. The device-mapper subsystem, originally aimed at raid devices and the like, has become near-ubitiquous, enabling once-esoteric technologies like LVM and LUKS/dm-crypt block-level encryption to become commonplace.

Without getting too detailed, I'll say that the device-mapper backend of Docker makes use of some of the newer parts of the lvm2 and device-mapper subsystems, specifically the "thin" provisioning and "thin external snapshots" capabilities. Let's fire up a scratch Vagrant machine, install Docker upstream, and stop the docker service:

$ vagrant init ubuntu/trusty64 A 'Vagrantfile' has been placed in this directory. You are now ready to 'vagrant up' your first virtual environment! Please read the comments in the Vagrantfile as well as documentation on 'vagrantup.com' for more information on using Vagrant. $ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu/trusty64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'ubuntu/trusty64' is up to date...
==> default: Setting the name of the VM: sandbox_default_1414021429729_97528 ==> default: Clearing any previously set forwarded ports...
==> default: Fixed port collision for 22 => 2222\. Now on port 2200.
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration... default: Adapter 1: nat
==> default: Forwarding ports... default: 22 => 2200 (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes... default: SSH address: 127.0.0.1:2200
default: SSH username: vagrant
default: SSH auth method: private key
default: Warning: Connection timeout. Retrying...
default: Warning: Remote connection disconnect. Retrying...
==> default: Machine booted and ready! ==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
default: /vagrant => /home/rsrchboy/work/CL/blog/docker-on-devicemapper/sandbox $ vagrant ssh Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-37-generic x86_64)
Documentation: https://help.ubuntu.com/ System information as of Wed Oct 22 23:44:04 UTC 2014 System load: 0.29 Processes: 88 Usage of /: 2.7% of 39.34GB Users logged in: 0 Memory usage: 18% IP address for eth0: 10.0.2.15 Swap usage: 0% Graph this data and manage this system at: https://landscape.canonical.com/ Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud 0 packages can be updated. 0 updates are security updates. [email protected]:~$ sudo su - [email protected]:~# curl -s https://get.docker.com/ubuntu/ | sh [...snip...]
The following NEW packages will be installed: aufs-tools cgroup-lite git git-man liberror-perl lxc-docker lxc-docker-1.3.0 0 upgraded, 7 newly installed, 0 to remove and 1 not upgraded. Need to get 7,670 kB of archives. After this operation, 36.2 MB of additional disk space will be used. [...snip...]

Re-Configuring Docker

From here we need to install the "lvm2" and "thin-provisioning-tools" packages -- otherwise things will fail in all sorts of mysterious ways:

[email protected]:~# apt-get install lvm2 thin-provisioning-tools Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: libdevmapper-event1.02.1 libreadline5 watershed The following NEW packages will be installed: libdevmapper-event1.02.1 libreadline5 lvm2 thin-provisioning-tools watershed 0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded. Need to get 1,239 kB of archives. After this operation, 8,863 kB of additional disk space will be used. [...snip...]


Next, docker itself needs to be told to use the device-mapper backend; this is as simple as ensuring that /etc/default/docker contains the line:

DOCKER_OPTS="--storage-driver=devicemapper"

...restart docker, and that should be all we need to do.

[email protected]:~# echo 'DOCKER_OPTS="--storage-driver=devicemapper"' >> /etc/default/docker [email protected]:~# service docker restart docker stop/waiting docker start/running, process 8857

Now, if we look under /dev/mapper, where device-mapper devices live, you'll note a couple special nodes, the control node and the docker storage pool:

[email protected]:~# ls -l /dev/mapper/ total 0 crw------- 1 root root 10, 236 Oct 22 23:43 control brw------- 1 root root 252, 0 Oct 22 23:52 docker-8:1-262146-pool

How to Use Different Docker Filesystem Backends

Let's try a simple approach. We'll pull down the latest Ubuntu image and fire up an interactive session:

[email protected]:~# docker pull ubuntu:latest ubuntu:latest: The image you are pulling has been verified 511136ea3c5a: Pull complete d497ad3926c8: Pull complete c5fcd5669fa5: Pull complete 49bb1c57a82c: Pull complete 67983a9b1599: Pull complete 88fba6f3d2d8: Pull complete eca7633ed783: Pull complete Status: Downloaded newer image for ubuntu:latest [email protected]:~# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE ubuntu latest eca7633ed783 42 hours ago 192.7 MB [email protected]:~# docker run -i ubuntu:latest

From our host, note what /dev/mapper looks like now:

[email protected]:~$ ls -l /dev/mapper total 0 crw------- 1 root root 10, 236 Oct 22 23:43 control brw------- 1 root root 252, 1 Oct 23 00:16 docker-8:1-262146-09e792e3f469e00fb1d9acde3ecf4b6d67bbd1729fd4d05bcc24eb2adb45fe18 brw------- 1 root root 252, 0 Oct 22 23:52 docker-8:1-262146-pool

Inside the container, if we look at what it considers the root filesystem to be, we can see it's using the snapshot we saw just above:

mount | head -1 /dev/mapper/docker-8:1-262146-09e792e3f469e00fb1d9acde3ecf4b6d67bbd1729fd4d05bcc24eb2adb45fe18 on / type ext4 (rw,relatime,discard,stripe=16,data=ordered)

We see our control node again, as well as the docker pool... and the device containing the root filesystem for our running container. Installing a mainline kernel is left for another blog post.