In each DeNA location like San Francisco or Santiago, game development is organized into multiple teams. Each one is independent, self-contained with game designers, back-end engineers, and front-end engineers. Each game needs a full-stack environment. For DeNA's operations team, the question is always how to get environments quickly to the game teams so they can chart their creative goals and change as they proceed.
In the world of game development where things change quickly, launch schedules are still the priority. Projects have several checkpoints before release. Games run separately as different entities where teams want to choose their own hardware and environment. DeNA values game developer creative freedom. That means although the architectural team reviews all the production game environment stacks, the operations team doesn’t force development teams to follow these configurations strictly. DeNA's IT operations team in San Francisco is small. Given the volume of game titles they push, providing game environment stacks custom fitted for every project checkpoint just doesn’t scale.
Also, the game teams include many junior developers who aren’t operations-savvy. DeNA wanted a more efficient way of giving deployment freedom in the cloud without that going completely uncontrolled and unsupervised.
Traditionally, DeNA had used RightScale and AWS EC2. They also hosted some of their hardware nodes in their own datacenters. And they continue to use Amazon Elastic Beanstalk for particular games that use PaaS. But Elastic Beanstalk presented challenges. It locked DeNA into a lot of the nuances like multiple user accounts on the machine, which is not good for audit trails and security. With RightScale, the operations team would set up the provisioning, but developers didn’t have any insight into that process. DeNA wanted a solution that did more than spin machines up or down because AWS already subsumed this functionality in its console. What they mainly wanted was to create a self-service environment
DeNA's instances are AMIs with Puppet based infrastructure configuration. They produce a vanilla Ubuntu 12.04 base image with Puppet classes on top of it. They required an easy way to build a library of scripts in a service catalog using Puppet. What they realized they needed was to operationalize and create the blueprint so that game teams could then run them on their desktops as well as in staging or production.
Game teams have varying needs and, therefore, need different technology stacks. DeNA's typical game environment stack runs the game code apart from the runtime libraries on Linux boxes. These run as Node.js-based applications with the mobile phone application as the front-end and MongoDB as the database.
Before Cloud Application Manager, if game teams needed a new environment, they opened a ticket with the operations team and waited a week or more to get it.
"When we have so many people who interact with us, we can’t possibly have everyone come through our pipeline for every need. The operations team has top priority tasks to scale systems and keep the production systems running. Our focus is heavily on improving game environments’ reliability and functionality. That’s why we need to give development teams control. Our goal was to allow game teams to self-service their environment needs quickly and also minimize the number of interactions with the small operations team in San Francisco."
"Now with Cloud Application Manager, our development teams are able to self-service custom virtual machines and environments for test, staging, and production in 5-8 minutes. That’s 30 times faster provisioning!
Developers can readily use the parameters of a box to set up custom environments. They can enter the version of Node.js they want or enter keys to SSH into the machine. Or choose between Redis or Memcached for caching, or integrate pieces of the stack into one box. Cloud Application Manager offers more flexibility for reuse. Game developers can modify production style environment boxes and reuse those same boxes to develop new game environments.
"We don’t have the time to figure out new technologies all the time. It’s a perfect opportunity for using a product like Cloud Application Manager. It helped us ramp up teams with our expansion in Santiago. It’s hard to create an atmosphere where people can grow while building the products, or grow from junior to senior engineers. A lot of the learning happens on their own time so creating an organic atmosphere for learning is vital. Cloud Application Manager allows for an organic way to transfer knowledge. We can point developers to an article on how to produce boxes and say, now go out and build your technologies."
When DeNA considered a RightScale alternative, collaboration ranked as an important consideration. In the end, the collaboration features (team workspaces and sharing) in Cloud Application Manager helped educate the junior software engineers faster. In less than six months of signing up with Cloud Application Manager, DeNA had most teams adopt the selfservice model of provisioning game environments. This adoption happened organically without DeNA management mandating a push.
“Cloud Application Manager helps the operations team stay in control of the bottom of the stack. We give game developers the keys to choosing their destinies, but we hold the skeleton key to everything.” — Jesse Sanford, Senior Systems Engineer and Lead Architect, DeNA
DeNA was spending 25%of their IT costs on RightScale because they charged by the number of instances. On the other hand, the per-user pricing in Cloud Application Manager made more cost-effective sense. To forecast user cost and relate that to game developers’ success was easier to predict and measure.
DeNa is a company with a DevOps perspective. Cloud Application Manager is a great DevOps tool for empowering developers. What we like most about it is that it’s super friendly. As an operations person, I don’t have to think much about what the developers are doing and in turn they don’t have to ask us. Rolling out self-service for easy access across independent teams was the main reason we went with Cloud Application Manager in the end.
"Another factor was the Cloud Application Manager team. They were very open to suggestions and made changes quickly in response to our challenges and concerns."
Cloud Application Manager helps the operations team stay in control of the bottom of the stack. This allows DeNA to give game developers the keys to choosing their destinies, but IT Operations holds the skeleton key to everything. They do that with common elements like creating sudoers files, which store root privileges. Or they work with things like systemd, a Linux service manager or they lock down parts of the system not appropriate for access, and so on.
DeNA (pronounced “D-N-A”) develops social games for iPhone and Android devices. Founded in 1999 by Tomoko Namba to delight and impact the world with new technologies, DeNA spans a global presence with offices from Tokyo to Santiago to San Francisco. Its latest title, Final Fantasy: Record Keeper, hit the 1 million download mark. DeNA also made news for its partnership with Nintendo to produce social games. To build dynamic, social games, DeNA relies on a vast team of talented engineers and the expertise of a small IT operations team in San Francisco. Jesse Sanford, senior systems engineer and lead architect at DeNA in San Francisco, talks about why they chose Cloud Application Manager to operationalize their software game development lifecycle.
— Jesse Sanford, Senior Systems Engineer and Lead Architect, DeNA