Managing logs for cloud infrastructure can be a daunting task. Fortunately, we have tools like Graylog to help us out. Graylog is useful for storing, searching, and analyzing logs from a multitude of different sources. One common log source is a Tomcat app running in a Docker container. However, getting logs from containers isn't always simple. Oftentimes, you have to spend extra time configuring other services like syslog or logstash to deal with log aggregation.

Thankfully, there is a faster way. The most recent release of Docker, v1.8.0, makes collecting and sending logs to Graylog much easier. Additionally, the logs are significantly enhanced with extra data such as container ID, container command, and configurable tags. These extra fields can be invaluable when troubleshooting an application problem. In this tutorial I will cover how to set up Docker to natively send Tomcat logs to Graylog. Using the ideas explained in this tutorial should help simplify your existing Tomcat container deployments and logging requirements. As an added bonus, the techniques discussed here can also be generalized to other services running in Docker.

Requirements

Docker version: >=1.8.0

Instructions

  1. Install Docker. Docker's native support for Graylog Extended Log Format (GELF) is new, and you will need to install the latest version of Docker for your host OS. For detailed instructions, please consult Docker Supported Installations. I'm using Ubuntu vivid and in order to install the latest version I simply run the following command:

    $ curl -sSL https://get.docker.com/ | sh
    
  2. Create a Java8 Dockerfile. We'll be using Dockerfiles to create two Docker images. The first is strictly Java8. In a later step, we'll use this image to build the Tomcat8 server. I like to create separate directories for each Dockerfile in case any helper scripts are needed during the image creation. First mkdir jdk8-oracle and then cd jdk8-oracle. Place the contents below into a file called Dockerfile. The Dockerfiles in this tutorial are available from GitHub.

    FROM ubuntu:vivid
    
    MAINTAINER Matthew Close "https://github.com/mclose"
    
    RUN apt-get -y install software-properties-common
    RUN add-apt-repository ppa:webupd8team/java
    RUN apt-get update && apt-get -y upgrade
    
    # accept oracle license
    RUN echo debconf shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
    RUN echo debconf shared/accepted-oracle-license-v1-1 seen true | /usr/bin/debconf-set-selections
    
    # install oracle java 8
    RUN apt-get -y install oracle-java8-installer && apt-get clean
    RUN update-alternatives --display java
    ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
    
  3. Build the image. This step will take a few minutes to complete if you haven't already pulled down the Ubuntu vivid container image.

    $ docker build -t mclose/jdk8-oracle .
    
  4. Create a Tomcat8 image. Similar to above, first mdkir tomcat8 and cd tomcat8. Then place the contents below into a Dockerfile. In order to use the latest version of Tomcat8, I'm installing tomcat manually as opposed to using apt-get which will install an older version. Basically, this Dockerfile creates a Tomcat user, downloads and installs the latest version of Tomcat8, and finally runs Tomcat. In order to keep the container running we tail -F the output of the logs. More importantly, by using the tail command, Tomcat logs are sent to stdout. The GELF log driver then picks up both stdout and stderr and sends them to Graylog. By tailing the catalina.log files we are able to direct Tomcat logs to Graylog. To generalize this to other applications besides Tomcat is simply a matter of tailing your service's log file.

    FROM mclose/jdk8-oracle
    MAINTAINER Matthew Close "https://github.com/mclose"
    
    RUN adduser --system --shell /bin/bash --gecos 'Tomcat user' --group --disabled-password --home /home/tomcat tomcat
    
    RUN wget http://www.us.apache.org/dist/tomcat/tomcat-8/v8.0.24/bin/apache-tomcat-8.0.24.tar.gz -O /tmp/tomcat.tar.gz
    RUN mkdir -p /usr/share/tomcat8
    RUN tar xzvf /tmp/tomcat.tar.gz -C /usr/share/tomcat8
    RUN rm /tmp/tomcat.tar.gz
    RUN ln -s /usr/share/tomcat8/apache-tomcat-8.0.24 /usr/share/tomcat
    RUN chown -R tomcat:tomcat /usr/share/tomcat8
    
    CMD /bin/su - tomcat -c /usr/share/tomcat/bin/startup.sh && tail -F /usr/share/tomcat/logs/catalina.out
    
  5. Build the image. This build should be much quicker than the previous one.

    $ docker build -t mclose/tomcat8 .
    
  6. Start a Graylog container. For the purpose of this tutorial, I'm going to use a Docker container to run Graylog. Since Docker is already installed this is the fastest way to test. However, if you already have a Graylog server configured, please feel free to use it. First pull then run the container. In this example, I'm exposing two ports. Port 9000 is used for Graylog administration. Port 12201/udp is used to receive GELF log entries. For more information, see Graylog and Docker.

    $ docker run -t -p 9000:9000 -p 12201:12201/udp graylog2/allinone
    
  7. Login into Graylog. http://localhost:9000. For the Docker container, the user is admin and the password is admin.

    Logging into Graylog

  8. Create the GELF UDP input in Graylog. From the main screen select System -> Inputs. In the dropdown on the next screen, select GELF UDP and click Launch New Input.

    GELF UDP

  9. Configure the GELF UDP input in Graylog. Simply give the input a Title and click the Launch button at the bottom of the window. There is no need to change any of the defaults for this tutorial.

    Launch GELF input

  10. Verify the Graylog input is running. Once launched, the input should now show as running in Graylog.

    Running GELF input

  11. Start a Tomcat8 container. There are a couple of important things at work in the docker command below: First -p 8080:8080 exposes the default Tomcat port so we can validate it is running. Next --log-driver=gelf selects GELF for logs, --log-opt gelf-address=udp://localhost:12201 sets the destination of the logs, and --log-opt gelf-tag="tomcat8 example" will set a useful tag in the Graylog entries. If you are using your own server, change localhost to the correct IP. For a complete list of Docker GELF options, please see Docker's Configuring Logging Drivers.

    $ docker run -t -p 8080:8080 --log-driver=gelf --log-opt gelf-address=udp://localhost:12201 --log-opt gelf-tag="tomcat8 example" mclose/tomcat8
    
  12. Verify Tomcat is running. Tomcat should have successfully started during the previous step. To verify go to http://localhost:8080.

    Verify tomcat8

  13. View Tomcat logs in Graylog. You should now see log entries for Tomcat within Graylog.

    Graylog showing tomcat logs

  14. View log detail. Click on one of the entires and you can see the amount of extra detail you get from the GELF driver. Not only can you see the log entry, but now it includes a wealth of extra data about the Docker container that sent the log.

    Graylog GELF entry detail

Conclusion

I hope this tutorial has convinced you that using the new Docker GELF drivers is well worth it. The extra data about Docker containers included in every log will help reduce troubleshooting time. Additionally, your Tomcat deployments should now be easier.