In each location like San Francisco or Santiago, game development is organized into multiple teams. Each one is independent, self-contained with game designers, back-end engineers, and front-end engineers. Each game needs a full-stack environment. For one gaming company's operations team, the question is always how to get environments quickly to the game teams so they can chart their creative goals and change as they proceed.
In the world of game development where things change quickly, launch schedules are still the priority. Projects have several checkpoints before release. Games run separately as different entities where teams want to choose their own hardware and environment. This company values game developer creative freedom. That means although the architectural team reviews all the production game environment stacks, the operations team doesn’t force development teams to follow these configurations strictly. The IT operations team in San Francisco is small. Given the volume of game titles they push, providing game environment stacks custom fitted for every project checkpoint just doesn’t scale.
Also, the game teams include many junior developers who aren’t operations-savvy. The company wanted a more efficient way of giving deployment freedom in the cloud without that going completely uncontrolled and unsupervised.
Traditionally, the company had used RightScale and AWS EC2. They also hosted some of their hardware nodes in their own datacenters. And they continue to use Amazon Elastic Beanstalk for particular games that use PaaS. But Elastic Beanstalk presented challenges. It locked the company into a lot of the nuances like multiple user accounts on the machine, which is not good for audit trails and security. With RightScale, the operations team would set up the provisioning, but developers didn’t have any insight into that process. The company wanted a solution that did more than spin machines up or down because AWS already subsumed this functionality in its console. What they mainly wanted was to create a self-service environment
Instances are AMIs with Puppet based infrastructure configuration. They produce a vanilla Ubuntu 12.04 base image with Puppet classes on top of it. They required an easy way to build a library of scripts in a service catalog using Puppet. What they realized they needed was to operationalize and create the blueprint so that game teams could then run them on their desktops as well as in staging or production.
Game teams have varying needs and, therefore, need different technology stacks. The company's typical game environment stack runs the game code apart from the runtime libraries on Linux boxes. These run as Node.js-based applications with the mobile phone application as the front-end and MongoDB as the database.
Before Cloud Application Manager, if game teams needed a new environment, they opened a ticket with the operations team and waited a week or more to get it.
"When we have so many people who interact with us, we can’t possibly have everyone come through our pipeline for every need. The operations team has top priority tasks to scale systems and keep the production systems running. Our focus is heavily on improving game environments’ reliability and functionality. That’s why we need to give development teams control. Our goal was to allow game teams to self-service their environment needs quickly and also minimize the number of interactions with the small operations team in San Francisco."
"Now with Cloud Application Manager, our development teams are able to self-service custom virtual machines and environments for test, staging, and production in 5-8 minutes. That’s 30 times faster provisioning!
Developers can readily use the parameters of a box to set up custom environments. They can enter the version of Node.js they want or enter keys to SSH into the machine. Or choose between Redis or Memcached for caching, or integrate pieces of the stack into one box. Cloud Application Manager offers more flexibility for reuse. Game developers can modify production style environment boxes and reuse those same boxes to develop new game environments.
"We don’t have the time to figure out new technologies all the time. It’s a perfect opportunity for using a product like Cloud Application Manager. It helped us ramp up teams with our expansion in Santiago. It’s hard to create an atmosphere where people can grow while building the products, or grow from junior to senior engineers. A lot of the learning happens on their own time so creating an organic atmosphere for learning is vital. Cloud Application Manager allows for an organic way to transfer knowledge. We can point developers to an article on how to produce boxes and say, now go out and build your technologies."
When the company considered a RightScale alternative, collaboration ranked as an important consideration. In the end, the collaboration features (team workspaces and sharing) in Cloud Application Manager helped educate the junior software engineers faster. In less than six months of signing up with Cloud Application Manager, the company had most teams adopt the selfservice model of provisioning game environments. This adoption happened organically without management mandating a push.
The company was spending 25% of their IT costs on RightScale because they charged by the number of instances. On the other hand, the per-user pricing in Cloud Application Manager made more cost-effective sense. To forecast user cost and relate that to game developers’ success was easier to predict and measure.
"Cloud Application Manager is a great DevOps tool for empowering developers. What we like most about it is that it’s super friendly. As an operations person, I don’t have to think much about what the developers are doing and in turn they don’t have to ask us. Rolling out self-service for easy access across independent teams was the main reason we went with Cloud Application Manager in the end.""
"Another factor was the Cloud Application Manager team. They were very open to suggestions and made changes quickly in response to our challenges and concerns."
Cloud Application Manager helps the operations team stay in control of the bottom of the stack. This allows the company to give game developers the keys to choosing their destinies, but IT Operations holds the skeleton key to everything. They do that with common elements like creating sudoers files, which store root privileges. Or they work with things like systemd, a Linux service manager or they lock down parts of the system not appropriate for access, and so on.