To service their global customer base, Cytobank must customize each site environment. Each site must have secure federated data access. At any given time, a site user requires high capacity VMs to fulfill client and backend application requests. Besides customer needs, Cytobank must provide demo site environments to their company scientists. And also provision QA environments that match the production site fidelity for their offshore development team in India to develop and test site changes. Cytobank needed an integrated orchestration platform to service the unique constraints of all their customer, demo, and QA site environments:
They wanted a system that would service ad hoc automated resource requests from customers, demo, and QA users in different time zones and geographies.
To scale operations in each site and manage a complex ecosystem of tools from a single place, they wanted an integrated platform with rich APIs. This platform should integrate with AWS, their cloud provider, as well as other complex tools in their site environment setup:
They wanted a platform to orchestrate provisioning, configuration management tasks, and then register systems in the appropriate site pools through API calls and scripts in a web of complex systems.
Federated access and huge data demands in each site meant they consumed a vast amount of VM resources. At scale, VM sprawl can add up to thousands of dollars very quickly. So they wanted a system to curb VM sprawl through the automated shutdown of idle resources.
How does a small IT operations team at Cytobank serve highly customized VMs on-demand for site customers, demo users, and the QA staff? All of whom are spread across geographies and multiple time zones. They wanted an easy self-service catalog for demo and QA users to spin up site environments on demand.
Robin Lee Powell, Director of IT Operations, says, “I can just give a set of people role-based access to the sites in Cloud Application Manager without giving direct SSH permission. The Cloud Application Manager web UI is easy to ramp up and add users. Demo and QA users can simply log in and self-serve environments at the click of a button.”
Cytobank chose Cloud Application Manager to integrate with a complex ecosystem and orchestrate site deployments automatically. Each site environment has a Tomcat application and web server, an NFS file server, and a MySQL database. All these servers run in AWS as Docker containers to fully isolate each site. “Cloud Application Manager bridges the gap between what’s on the AMI and what needs to happen to configure the machine as a fully compliant member of its pool” adds Robin.
With Cloud Application Manager, the Cytobank IT operations team saves time on writing scripts and managing complex custom environment configurations for each site. Do they maintain 40 different AMIs for each site environment? No. Instead, they define the site stack as boxes in Cloud Application Manager. By defining the Tomcat, NFS server, MySQL, and Puppet boxes once, Cytobank reuses the same base boxes to launch custom configured site environments for production, demo, and QA users alike.
A top requirement for them was to regulate resources and save on hosting costs. But the highly computational nature of their application involving huge sets of data as well as federated site access meant that Cytobank incurred significant costs to maintain these environments. So a system whose APIs would respond to a sophisticated decision-making engine was imperative. It would automatically spin up or shut down idle resources and environments based on live usage metrics.
What does Cytobank do now to save costs? They spin up and scale or shutdown resources in site environments with scripts and Cloud Application Manager API integration. In any given site, the MySQL docker instance runs scripts that monitor active site usage in the MySQL database. When usage demands more resources, the scripts trigger Cloud Application Manager to launch the box defined stack onto AWS Docker environments. As soon as the machines boot up with the custom AMI, Cloud Application Manager registers them in Puppet, which configures them. Once the Puppet scripts run successfully, Cloud Application Manager registers them in the right AWS Elastic load balancing pool in the right site using the configuration data from the MySQL server.
Finally, when resources are idle, the scripts again trigger Cloud Application Manager APIs to shut them down. Robin explains, “If an environment is not in use for 3 hours, we shut them down. I prefer Cloud Application Manager APIs because only they tell me accurately which machine goes up or down, what state they’re in, or how long a machine has been up. This information is critical for us to be able to learn which resources are unused so we can automatically shut them off.“
“Choosing Cloud Application Manager has been one of the best decisions at the right time,” says Robin. They see value from the following benefits:
Cytobank QA and demo users can self-serve environments from an intuitive Cloud Application Manager web interface in minutes rather than make ad hoc requests to the operations team at odd hours given the time zone differences. This sort of self-service saves the operations team a lot of pages and on-call time that would otherwise require at least one person full-time to handle.
Cloud Application Manager automation saves Cytobank three months of operations effort and the cost of hiring additional personnel. Due to the Cloud Application Manager API integration and orchestration with AWS, Puppet, and their decision-making engine, they no longer need to do things manually. No more writing scripts manually to manage complex custom configuration for each customer, demo, and QA site environment.
Robin notes, “Cloud Application Manager is the one place where you can see everything. All the scripts are there to go in and figure out what went wrong. Even if something does go wrong, people other than me know where to look and fix because they’re used to serving up environments from Cloud Application Manager.” He goes on to add, “It’s one place where you’re not worried about bleed-over between the different environments. I can decide who can see what. Cloud Application Manager access control helps us address security concerns in the company as to who can see what environments.”
“The auto shut down of QA environments alone results in a minimum of $10,000 savings monthly. This is significant savings for us. It saves us from consuming about 50 additional machine resources a month,” says Robin.
Cytobank is a SaaS service for biopharma labs, university research institutions, academic research facilities like Stanford and Irish Labs as well as independent researchers. Its customers span globally across the US, Japan, Germany, Australia, and China. Researchers and clinicians study cancer and conduct other medical research then record the data in Cytobank. They organize, analyze, share, and visualize large sets of single-cell cytometry data in Cytobank and even perform complicated math on them. Simply put, Cytobank takes very large and complicated single-cell research data and turns it into meaningful, useful information.
— Robin Lee Powell, Director of IT Operations, Cytobank