About Chris Molozian: Chris is a founder and CEO of Heroic Labs, a game development company that has an API that allows game developers to develop, customize, and add innovative and interactive capabilities directly into games quickly and easily without forcing developers to deal with and navigate a server's backend. Heroic Labs builds social infrastructure for developers and is focused on social and real-time experiences for the games industry. The following interview is a tech-driven chat with Chris Molozian about online game development.

Erlang and Rust are your favorite languages. Those are obscure choices, especially to a game developer. What's the story behind that?

For business development on the platform for Heroic Labs, the main reason we chose Erlang is that as a small team, we want to be able to build scalable and resilient systems without requiring a larger Operations and DevOps team. We wanted to be able to build a system that was able to have the level of availability guarantees we were looking for without requiring us to have the kind of Operations teams that, in my experience, are more necessary with other programming languages. Erlang’s features – supervision trees, restart strategies, and the ability to monitor process ID's – tie together in the way the language is designed, to enable you to build systems that I describe as self-healing. This self-healing is evident in the ability for a system to transparently recover from failures, network timeouts, disk failures, all different kinds of transient failures that you tend to get when you’re running inside cloud infrastructure on top of infrastructure-as-a-service (IaaS) providers like AWS. Erlang allows us to more efficiently manage those kinds of restart strategies and transient errors that can turn into cascading system failures.

Rust was more of a personal choice for us to be able to experiment, with a degree of safety, at a systems level when it comes to getting the kind of performance requirements necessary from things like dynamic language runtimes or integration with C-based libraries for things like v8 for javascript runtimes or lure. We wanted to wrap them in a safer language that still enables us to achieve the kind of memory safety guarantees you typically get with managed languages, where they have some kind of garbage collector or managed object lifetime strategy. We don’t use any Rust code in production yet, but it’s an area where I’d like us to get closer to BareMetal, without constantly having that deep concern that if we don’t have a huge engineering team with an advanced knowledge of C++/C we may be leaving behind memory that can be double free or introducing memory errors, memory leaks, and security vulnerabilities into our code because memory management strategies can get very complex as code bases get larger and larger over time.

That’s what I think Rust is very good at – giving you that BareMetal guarantee of performance while introducing lifetime concepts around managing objects and states between threads and system-level communications. Rust has simplified C++ to the right extent. The lifetime controls you get over data and the way you share data between threads does introduce a more cognitive load for the engineer (you need to understand how this data is being owned by different threads in the system), but overall, and as a result of it, you get more explicit control of how the code communicates together and reduces the likelihood of errors creeping into the code.

How do you use Orchestrate Events?

Events is one of our smaller use-cases – intended for us to enable an audit trail. We use it to build up time-based audit trails on some developer actions performed inside the configuration of our dashboard. It was easy for us to build. It was a no-brainer to use Orchestrate for that part of our system.

How do you use key value listings and filters (filter queries)?

We use that to namespace data that belongs to different games within our system. We put all data into a particular collection and then we filter that data out so that specific games that own certain elements of data only have access to those records within their game. It’s a way for us to build a multi-tenant storage on top of a collection system like Orchestrate.

How do you use search and how do you decide when to use it instead of key value listings?

Search is our most widely-used feature. It’s embedded inside of our game API in what we call shared storage. Shared storage is a way for players to share objects (data) between each other in the game. Take a game like BoomBeach from SuperCell; within the game players build up an army and those armies defend their islands and then other players that are in the vicinity of those islands are able to compete in some kind of combat scenario – attack the islands, attack the armies, etc.. So to power that, we enable game studios to say that the particular data being stored into our shared storage API is read-only accessible to all other players in the game. What happens is the game client can retrieve a collection of criteria based on properties that they want to match with opponents in the vicinity of a game through a search query.

This allows for optimal opponent matching, i.e. someone can be matched with another opponent based on who has close to the same experience or similar weaponry. It can also allow gamers to play against old records they set for themselves or, for example, other players set in a racing game. Anything that can be stored can be retrieved again for the most relevant and ideal playing experience – this also helps us to establish a connection between players without requiring a real-time/live connection.

A search engine is built to give you relevancy, and yet not a lot of the industry invests in incorporating it into their systems; probably because this kind of search infrastructure is so complicated to run, which is why it’s easier to do it on top of Orchestrate.

You're the CEO – how often do you write code?

Not as much as I’d like. I do a lot of the tinkering and experimenting – with new game engines, new APIs – seeing if we can drive some new innovation in API design from those areas. I probably spend 30-40% of my time writing code, just because I end up needing to do everything else – business development, marketing, sales...everything.

What's your development setup look like?

SD for service discovery (a tool from Google) allows services in our infrastructure to publish themselves for other parts of the system to find and communicate with.

Our system has four major types that have a collection of services:

  • The Account System: handles player accounts, social login (integration with all the social networks), and upcoming work on things like social graph.

  • The Game API: the core set of APIs that are available for game developers to use from their game clients – leaderboards, cloud storage, achievements, and everything that also communicates with Orchestrate.

  • The Management System: everything that is configured through our developer dashboard is managed in the management code and servers and then gets mirrored into the rest of our services so they can have a cache copy of it, to reduce latencies and reduce dependencies between services.

  • The Cloud Code: JavaScript execution service that we’ve set up. It has a collection of services that handle untrusted code execution so developers can upload scripts and manage scripts through Github, which they can access and execute through the game client on our API. We have that as an isolated part of the system – it's effectively an execution environment that uses a combination of Java and Erlang to execute these scripts within a runtime and enable them to make things like HTTP requests, do data validation, do business rules (if you need rules around which player wins after a match is complete) and execute any other custom behavior that a game developer needs. We have developers who’ve built custom game tournaments or gifting and rewards through the code base and everything in between.

All of these systems interact with each other as they need to.

OK, let's switch gears...how do you make an Old Fashioned?

Sugar syrup/sugar water, orange rind, some kind of whiskey (rye of some kind), some ice, and some bitters. Still my favorite drink.

What's Next?

Sign up for our Developer-focused newsletter CODE. Designed hands-on by developers, for developers. Keep up to date on topics of interest: tutorials, tips and tricks, and community building events.

CenturyLink Cloud – We’re a different kind of cloud provider – let us show you why.