Note: This post was written on the Orchestrate blog before CenturyLink acquired Orchestrate

Ever since we launched Orchestrate’s Enterprise offering, we’ve gotten a lot of interest in running Orchestrate on cloud platforms other than AWS. One of the most often requested and interesting is CenturyLink Cloud. CenturyLink Cloud is a complete IaaS offering that competes head on with Amazon Web Services.

Given the interest in CenturyLink, we decided to put the platform to the test. We built a full Orchestrate environment on CenturyLink and ran our standard suite of benchmarks.

Methodology

For this benchmark we used a customized version of basho_bench. We use basho_bench because it collects the most relevant statistics, can push a ton of load, and is easy to extend.

Each of the runs was executed from a single server with 8 vCPUs and 17GB RAM. Before the initial run, we performed a data load of 1 million keys into a single collection with values around 1KB. Before each run, a brief pre-warm was done to prime the load balancer.

It’s worth mentioning before we get started that increasing the number of load generating servers would increase total throughput, but we did not measure by how much in these tests. We were simply trying to get a gauge of how well a relatively basic benchmark would perform against our deployment in a new cloud environment.

Get/Put/Delete

This benchmark used 75 concurrent workers doing get, puts, and deletes at a 50:50:1 ratio. Keys were selected from the million initially generated keys based on a pareto distribution.

Throughput 1

As you can see from the results, we are able to push 7800 req/sec from a single node without any difficulty.

The latencies were as follows:

mean median 95th percentile
get 8.4ms 7.4ms 16.5ms
put 10.9ms 10.9ms 21.0ms
delete 10.7ms 9.3ms 20.5ms

The raw data for this run is available here.

Production Mix

This benchmark used 60 concurrent workers doing a workload mix based on what we actually see in production. It performs gets, puts, deletes, list, and search operations at a 20:51:1:9:19 ratio. Keys were selected from the million initially generated keys based on a pareto distribution.

Throughput 2

This test has a more varied and complex workload, so we expect lower throughput as search and list return multiple objects. Even so, we were able to push nearly 5000 reqs/sec.

The latencies were as follows:

mean median 95th percentile
get 6.9ms 6.9ms 13.9ms
put 13.2ms 9.5ms 37.7ms
delete 12.8ms 9.3ms 36.8ms
list 25.5ms 23.0ms 46.5ms
search 9.8ms 6.8ms 32.5ms

We do see a strange pattern in the results. At regular intervals we see throughput drop and latencies spike. This is likely due to a performance bottleneck that has been since resolved. To verify API keys and map user collection names to our internal names, we must query a system external to our core database API. While the results are cached, when the cache expires we much reach back out and request this data. These requests tend to add 100ms or more of latency. Since we identified this bottleneck, we’ve designed a better caching scheme that requires fewer lookups and thus eliminating the performance drop-off we observed in this benchmark.

The raw data for this run is available here.

Low Throughput/Low Latency

This test is meant to be a measure of latency alone. By reducing the amount of load, we can measure latency under ideal conditions. For this, we use 10 workers with a get to put ratio of 1:1. We also reduced the size of the values generated to 133KB.

Throughput 3

Throughput was not something we were testing in this run. We did manage 1000 ops/sec even with so few workers.

The latencies were as follows:

mean median 95th percentile
get 10.1ms 8.3ms 25.9ms
put 10.9ms 8.4ms 33.1ms

The raw data for this run is available here.

Thoughts

Overall, we were quite happy with CenturyLink Cloud. The web based GUI is one of the easiest to use in the industry. The pricing is competitive relative to other clouds. Finally, when looking at performance as compared to AWS, we have our benchmarks from late last year.

Comparing the two results, we see that CenturyLink performs better than AWS in these benchmarks. While, the tests run were not exactly the same, we can compare the measured latencies. We see that CenturyLink performed better on nearly every measure.

If you are interested in having a dedicated Orchestrate cluster on CenturyLink Cloud, contact us here.