Data Visualization


I work as a Product Owner on the Advanced Storage team at CenturyLink Cloud®. My totally awesome team is comprised of 7 developers, an analyst (with dev chops) named Dan, and myself. One of the things I really enjoy about being a product owner at CenturyLink Cloud is that I have the autonomy to solve my own problems. I have access to, and I am encouraged to use, all the capabilities of our platform. It also helps that I have a development background, which gives me a desire to learn tech and use it to do fun things.

My team has been busy rolling out our next generation Object Storage product to Canada and the Eastern US. After we recently launched our product for the first time, Dan and I took some time to reflect on what tooling we needed in order to track against our business goals. We realized that we needed better visibility into how customers use our product. What customers and storage buckets are growing? At what rate are they growing? What customers are deleting their data? We desperately needed business intelligence on our product.

At CenturyLink Cloud, our internal Analytics team developed an ELK stack for our product teams. My team logs a number of things, including all customer API access to our Object Storage cluster, which we then plug-in to Kibana dashboards that allow us to use this data (think API calls per second, Delete operations vs Get vs Put vs Post, etc.) to make tactical decisions about our product. However, what we didn’t have was the data and visualizations we needed for our long-term business goals. Dan and I wanted to take this task on with as little developer time as possible. We wanted to own it.

What We Did - MVP

The team already had an auditor job that regularly queried our Object Storage system and reported bucket sizes to the platform billing system. Step one in our process was having our Devs modify our auditor to log the data to the Analytics team's Kafka service. This provided immediate value, as Dan and I were able to visualize the data using all of the powers of Kibana.

After a while, we began to bump up against limitations in Kibana. We wanted to do some calculations and visualizations that Kibana didn't support. Also, the data in the backing Elastic Search datastore was expiring after only a few weeks. We needed something bigger and more permanent.

What We Did - Phase 2

The first issue we needed to solve was the data expiration problem. We could have modified our auditor to store the data in a database. However, modifying that auditor is not easy and it tends to have a long turnaround time. Our next idea was to just pull the data from Elastic Search on some interval and push the data into long-term storage. We could have rolled our own database on the platform, but then we would have had to maintain it. We could also have used our Managed DB product, but that seemed like overkill for what would have been one or two DB tables. Since we wanted to keep the support/maintenance efforts to an absolute minimum, we decided Orchestrate looked like a perfect fit, as it required zero infrastructure and maintenance.

Now that we knew where we wanted to park the data, we had to figure out how to get it there. I'd had some exposure to Ruby and it seemed like a great, light language to solve this problem. On top of that, Orchestrate already had a Ruby Client Library. Just a few sweet lines of Ruby later, we were able to pull the data from Elastic Search and pump it into Orchestrate. The data moving process was set to run on an interval in order to make it hands-off as much as possible going forward.

The next step was to select a presentation stack for the data. Again, our goal was to keep the support time to an absolute minimum. AppFog made a ton of sense for this application. We were a little apprehensive, having never used AppFog before, but felt it would be a great learning experience. Having been bitten by the Ruby bug, and knowing that AppFog supported Ruby, we had our language picked. Also, Ruby on Rails would provide us a short path to having a functional app up and running.

Within a matter of minutes, we followed the AppFog KBs for setting up and got ready to push. We generated a default Rails app and pushed it up to AppFog. It worked! We were shocked how easy it was for a Ruby on Rails novice and a total AppFog newbie to get an app up and running. Fast-forwarding a little, we built Rails APIs, which are comprised of Cyclops and a JavaScript charting library called Plotly, to stream the data to the view. Basically, that's the platform recipe we used to solved our conundrum. We were excited to begin using the solution!

Metrics on Display

The result of pulling together all of the Platform technology is a Metrics Dashboard that provides my team with business intelligence for our product. It can tell us:

  • How our buckets, accounts, and total storage are growing over time
  • What customers are growing and which are shrinking
  • Which locations are growing faster than others
  • How much of our storage is used by internal teams vs external customers
  • What is the usage pattern of an individual account/bucket

Pies Charts

Other features: (Which we can't display in detail, but you get the idea.)

  • The top 100 buckets/accounts at any given time
  • Compare account/bucket storage changes for two dates

Time Chart

Time Chart Details

We now have the business tools we need for our product. We can reach out to new customers and welcome them to our product. We can reach out to customers who are deleting their data or buckets and find out why. This is huge for us.

Power of the Platform

The Metrics Dashboard shows off the tooling power that Data Analytics, AppFog, and Orchestrate bring to the table. Almost all of these technologies were new to Dan and I. Even so, we were able to deploy a scalable app without putting much thought into the operations behind it, including infrastructure the app runs on. That is awesome because it leaves us more time to focus on our product, which is what we really enjoy.

Thanks for reading, Phil Jensen, Product Owner - Advanced Storage

Platform CenturyLink

Want to learn more about our complete Enterprise Cloud Platform? Designed for your business needs today and tomorrow, the CenturyLink Platform is reliable, secure, robust, and global. As we like to say – it's your platform in motion!

Don't have an account on CenturyLink Cloud? No problem. Get started with CenturyLink Cloud for free and receive a healthy credit toward any of our products or services.