I just published a new version of the Orchestrate Gem, version 0.9, that provides a ruby-like interface for accessing Orchestrate’s API. Since the initial gem announcement, I’ve mapped Orchestrate’s data types and API semantics to ruby classes. I won’t cover everything here, because that’s in the rdoc. Instead I want to give an overview of the basics, and inspire you to check out Orchestrate’s Database-as-a-Service and the Ruby client by showing how easy it can be to get started.

If you’re already using the Orchestrate ruby gem, you won’t need to change any of your code. Very little has changed with Orchestrate::Client since its release, and this new interface is built on top of it.

To begin, you’ll need the Orchestrate gem. If your ruby installation is managed through rbenv or rvm you can omit sudo from the below commands:

sudo gem install orchestrate

Applications, Collections, and Keys, oh my!

First you have to create an instance of Orchestrate::Application. Provide your API key and optionally configure the HTTP client, identically to the Orchestrate::Client interface:

require 'typhoeus'
@app = Orchestrate::Application.new(ENV['ORCHESTRATE_API_KEY'])

Then lets explore creating KeyValue pairs:

@posts = @app[:posts]
# Let's reduce our typing by turning the collection into its own object.

@posts << { state: "draft", content: "blah blah blah" }
# this will auto-create the key and return a KeyValue object

@posts['orchestrate-gem-announcement'] = { state: "published", content: "Today I announce..." }
# this will set the value of the key 'orchestrate-gem-announcement'
# to the value on the right-hand side of `=`.  
# Per ruby conventions, it returns the RHS value.

@posts.create('orchestrate-gem-update', { state: "draft", "content": "I just published..." })
# This will set the value of the key to the provided
# value using 'Put-If-Absent' API semantics, and
# returns a KeyValue object.

@posts.set('orchestrate-gem-update', { state: "published", "content": "I just published..." })
# This will set the value of the key to the provided
# value, and does not provide a condition.

# Returns an unsaved KeyValue item with the given key.

The collection implements Enumerable to iterate over the keys using the List API. You can retrieve all of them or just a subset by key:

@working_post = @posts['orchestrate-gem-update']
# retrieves a single KeyValue item by key

# returns nil instead of raising the `NotFound` error.

all_posts = @posts.to_a
# Retrieves all posts, traversing the API's pagination,
# and returns them all as an array.

some_posts_starting_with_o = @posts.start('o').before('p').take(5)
# Queries the List API endpoint for posts with a first
# character between o and p, and limits the number of
# results returned to 5.

To retrieve a large number of KeyValue items and process them in smaller chunks, you can use Lazy Enumerables in Ruby 2.0 and later:

@posts.lazy.each do |post|
  File.open("#{post[:key]}.json", "w") {|f| f << post.to_json }

In this example, it writes each item in the posts collection to disk, but requests them as needed so no more than a hundred should be in memory at a time.

KeyValue: Wrapping Your Data With Persistence Logic

All of the items returned from @posts above are instances of the Orchestrate::KeyValue class, the main type of data in Orchestrate. Rather than create a bunch of heavyweight tools for meta-programming, I created a simple class for managing the responsibility of persisting data to Orchestrate.

For those looking for acts_as_orchestrate_object, I'll show how to use your own classes at the end of this section.

First let's look at one of these KeyValue items:

post = @posts['orchestrate-gem-update']

post.loaded? # => true
# this would be false if we created the KeyValue
# object with Collection#build or Collection#stub
post.last_request_time # => 2014-08-31 16:33:08 -0700
# When was this retrieved from Orchestrate?

post.collection # => @posts
post.collection_name # => 'posts'
post.key # => 'orchestrate-gem-update'
post.ref # => 'ec43c3a47d97b6f7'
post.value # => { "state" => "published", "content" => "I just published..." }
post[:state] # => "published"

The contents of post.value are the JSON-structure returned from Orchestrate, which defaults to a ruby Hash Since Hash implements #[] and #[]= accessors, the KeyValue implements the same accessors against the Value, while stringifying the key of the value object specified.

You can update the value and then persist the KeyValue as well:

post[:state] = "draft"
post.value # => { "state" => "draft", "content" => "I just published..." }
post.save # true

Because the post object is aware of the current ref from when it was loaded, it will use "Put-If-Match" semantics to tell Orchestrate that the key should only be updated if it hasn't been since the object was loaded.

Alternately, you could update the value and perform #save!. This raises an error instead of returning false should the save fail.

post.update!({state: "draft"})

Refs are Forever

Setting a key's value from the collection (@posts) will override an existing value. Updating the value from a loaded KeyValue object will use different semantics in the Orchestrate API to only update the value, if it hasn't changed since the KeyValue object was loaded. This leverages a cool feature of Orchestrate's data model called refs, a specific immutable value that has been assigned to a key.

When you update the value for a key in Orchestrate, it keeps the previous values that have been assigned. When you delete a key in Orchestrate, you're actually setting its value to null. You can retrieve all the previous values for a key, and you can purge all of the values as well.

Given a KeyValue object, you can access its refs easily:

post.refs.map {|ref| [ ref.ref, ref.reftime ]}
# returns an array of arrays with ref identifiers
# and the timestamps at which they were created

ref = post.refs.offset(1).take(1).first
# returns the most recent value that is not the current one
ref.kind_of?(Orchestrate::KeyValue) # => true
ref.class # => Orchestrate::Ref
ref.archival? # => true
ref.tombstone? # => false
# When a KeyValue is deleted normally, its value is
# set to null.  The reflist indicates this with a tombstone marker.

# retrieves a ref by its identifier

Because the Ref class inherits from KeyValue, it behaves the same. The save operation uses "Put-If-Match" semantics, so if you attempt to save it and the ref is outdated, saving will fail.

Events Happen

Orchestrate's Events are a way to associate time-ordered data with a key. They have a "type" and are enumerable by time range, and identified by a timestamp and ordinal value.

Let's say we wanted to implement comments for our posts:

post = @posts.stub(request.body.post_key)
# This will return a KeyValue that is not "loaded", so you can access its refs, events or relations without having to make a round-trip to Orchestrate first.

event = post.events[:comments] << request.body.comment.merge({ip: request.ip})
event.timestamp # => 1409545018685
event.time # => 2014-08-31 21:17:09 -0700
event.ordinal # => 10032
event.value # => { "body" => "wow what a long post",
            #      "name" => "Trolly McTrollerson", ... }

This operation returns an Orchestrate::Event which, like KeyValue, you can save, update, delete it, et cetera.

You can also enumerate over all events or a subset of them with Enumerable:

comments = post.events[:comments] # returns an `Orchestrate::EventType`
comments.after(Time.now - 24 * 3600).end(Time.now).take(25)
# Returns the up to 25 comments from the past day, newest first

comments.lazy.map{|comment| [comment.time, comment["name"]] }.to_a
# Will return an array of arrays time and commenter name,
# fetching results 100 at a time

The Graph to Your Heart

Finally, we have graph relations, Orchestrate's answer to relational joins. You can create directed relations between KeyValue objects, even across collections, and then easily traverse them to any depth.

Continuing with our blog example:

author = @app[:users][:mattly]

author.relations[:authored] << post
post.relations[:authors] << author
# Each of these lines will create a reation of the type in
# brackets with the KV item being pushed into it

post.relations[:similar_posts].push(:posts, 'some-other-post')
# If you don't have the related Key Value item instantiated,
# you can use the collection name and key instead

Once you've created the relations, walking them is straightforward:

# returns all the authors for the post, instantiated as KeyValues

# a bit more complicated, asks about all the authors of the post,
# all the posts they've authored, all the similar posts to
# those posts, and finally all of the authors of those similar posts.

Orchestrate returns all the related items at once, so while the relations[] traversal object is an Enumerable, there's no benefit to using it lazily.

In Conclusion

All of this functionality was built atop the main client that I had described in the original release. I said in that release announcement that I wanted to provide a first-class ruby experience of the Orchestrate API, but first I wanted to have a solid and flexible underlying base client.

I didn't quite get to the cursors, but instead created something perhaps a bit more flexible by using Enumerable for iterating over lists of KeyValues, Refs, and Events. I've tried to keep the code straightforward, because if you do need to troubleshoot a problem, delving through magic code is the last thing you want to do.

One of my goals with this work was to create something on which to build an ActiveRecord adapter. Another goal was to avoid the set of problems commonly associated with Object/Relational Mapping. I'd love to hear your feedback!