DevOps is a growing phenomenon amongst companies. They want to develop software faster, with the flexibility to adapt quickly to changes as they arise. Strategically, companies vary in the processes used to implement the DevOps model. This post, based on Richard Seroter's OpenStack summit talk, will provide a little background on DevOps and describe what it took to bring the approach to CenturyLink.

What is DevOps?

By definition DevOps is a term for a group of concepts that, while not all new, have catalyzed into a movement emerging from the collision of two major related trends. The first trend, “agile system administration” or “agile operations”, sprang from applying newer Agile and Lean approaches to software engineering and operations work. The second trend was to form a routine that fostered a greater appreciation for collaboration between development and operations teams throughout all stages of the software development life cycle. This stems, in part, from the experience of creating and operating a service at scale, and how important operations have become in our increasingly service-oriented world.

Team collaboration, photo by Barrett Hall

The DevOps life cycle spawned from Agile, where there are four core iterations, or "cycles," that create the complete life cycle. These are Backlog, Development, Collaboration, and Production. The key element of success is adapting the model so that these cycles are fluid, the process time is decreased, and the team dynamics (along with automation) allow for a continuous delivery cycle. At CenturyLink Cloud, we had core business strategies that we wanted to ensure were cultivated both continuously and efficiently, and that is where our journey into DevOps began.

How and Why DevOps at CenturyLink Cloud

We wanted to implement a strategy that allowed for greater overall velocity. Doing DevOps just for the sake of it is meaningless. Implementing a system that yields the desired outcome while maintaining the core fabric of your company, well that made sense.

Why we felt our company and customers would benefit from the change:

  • CenturyLink acquired Tier 3. It became evident very soon how well the model worked in that environment. The catch was could we “bottle” that up, scale it to a larger group, and reap the same output of velocity?
  • Adapt a work flow model that allowed for our Core Values to be embedded into the daily process.
  • We wanted to stay with the up-tick in the competitive market. To do so we had to migrate to a method where we optimize our business goals on a daily basis:

    1. Ship code faster and more efficiently.
    2. Deliver scalable services and products to market with a continuous delivery cycle.
    3. Actively responsive and agile when issues arise.
    4. Have the ability to launch new features all the time.
    5. Provide both world-class uptime and competitive pricing.

So, we had our Why. Then came the hard part, the How. Implementing a lean culture into a large enterprise was just one of the challenges. Incremental changes were made to ease the progression of the shift.

How we approached change within the organization:

  • Buy-in — For any drastic shift in vision or work flow for a company or project, buy-in is essential. It needs to be a trickle-up buy-in, different from most implementations of change in the corporate world. To allow for a smooth transition, the talent that was going to be a part of this culture shift needed to buy in to the long term vision and goal. In this case that meant the feature team members: the Engineers, product analysts and owners, and the operations team.
  • Org Chart and Team Structure Re-Org — Before we began our path to DevOps, our Org Chart was matrixed and was not very scalable. It was easy to lose focus on the tasks at hand, the productivity levels, and, most importantly, customer needs and expectations. Org chart before DevOps Senior leadership came together and devised a plan on how to flatten the org, and re-purpose the high amount of over-head into truly productive roles. The team structure was altered as well. They broke down silos and created feature teams, each with a franchise model and a product or service to produce and ultimately own. Here's how the chart looks after of the restructure: Org chart after DevOps
  • Tools — As your work flow evolves, so should your tools. Various tool sets were implemented to make Communication, Collaboration, and the cycle of each team visible and measurable. Below are just a few:

    • Trello — Easy tool to utilize, to see work in progress. Each feature team has a board that has backlog and current sprints. Allows for transparency, accountability, and simple tasking.
    • GitHub — Code and knowledge base, all open source.
    • Slack — Killing email. Breeds a collaborative culture and integrates with Trello and GitHub.
    • ZenDesk — support ticketing system.
    • WebEx/Skype — Meetings are infrequent, but when needed, generally due to a remote team member inclusion, these are the top tools of choice.
  • Workspace — For the continuous delivery cycle to thrive, the environment must support it. We did this with:

    • Team rooms — Each feature team is housed in team rooms where the conversation is relevant to the cycle. Desks are functional for standing or sitting, tailored for collaboration on the fly, stand-ups, and deployment. It is crucial for the workspace to be conducive for the team to easily pivot between developing and collaboration. Some customs that were implemented to foster both comradery and respect for the work space was no phone calls in the team room and lunch is not allowed in the team room.
    • Common space — A large area in the middle where everyone eats together, encouraging teams to cross pollinate.
    • Executive engagement — There are few offices in the Bellevue site. The culture is that executives are engaging with the team or customers. They are often contributing and collaborating with the teams in the team rooms.
    • Learning and community — A space designed to show case releases, doing tutorials, to collaborate with other teams on vested interests and where we hold community events.

If you’re in Bellevue come take a tour of our facility or check it out here.

We are done here, right? We successfully implemented DevOps into our culture. Not quite. DevOps is something that you have to cultivate. Its inherent nature is based off learning from usage. That is the stage we are currently in at CenturyLink Cloud. We are continuously striving to optimize, streamline, and automate processes. There are bumps along the way. Every retro session unveils a lesson learned, a key to a simpler path, or the future iterations of how we can evolve with this process as our company grows and transitions. We will share our journey here, at the Developer Center through tutorials, blog posts, knowledge base articles, and developer tools like SDKs. The DevOps path for us has been laid. Now we have to navigate it, learn from it, and adjust as needed in a vested interest towards either Travesty or Triumph.