I took a quick break from working on the Orchestrate Ruby Client to answer a challenge from Orchestrate COO Matt Heizenroder: could I port Panic‘s PunchClock‘s ruby-based server component from PostgreSQL to Orchestrate and write a blog post about it? In 48 hours?

Of course I said yes. We’ve talked in the past about how iBeacons are an interesting technology, I’m always looking for more excuses to do iOS development, and it’s been a while since I’ve had a frantic deadline to race towards. Cue the coffee and adrenaline!

You can read more about the app at Panic’s blog post, but gist of it is that it combines background iOS location services with a web app that maintains a list of people with the compiled & configured iPhone app who are within a certain radius of some pre-defined geo-coordinates. Basically, “who’s in the office?” You can ‘watch’ someone to get a push notification when they arrive or leave, and you can send a message to anyone who is “in”, such as “hey, I’m locked out”.

This post is long, mostly because I’ve tried to avoid assuming too much knowledge about either Orchestrate, Ruby, or iOS development. To follow along, you’ll need the following:

  • An Orchestrate Account (free, signup here)
  • A Heroku Account (free)
  • A ZeroPush Account (free)
  • A Mac (not free) running OS X, preferably Mavericks
  • Xcode (free)
  • An iOS Dev Center account ($99/year)
  • An iPhone or iPod touch running iOS 7, possibly 7.1.2

I’ll provide you with instructions for signing up for services as needed.

The server-side component is a fairly simple Sinatra app and its database schema was two simple collections: one to represent people and one to define relationships between people. It’s a great fit for Orchestrate.

Getting Started

The first thing I like to do with any project like this, before modifying anything, is get everything setup and working before changing anything. To figure out how to build things, deploy them, and verify everything works as expected, before I set about breaking it. So here goes:

Setting up the Server

I forked Panic’s PunchClock Server repository to my own Github, and cloned it down.

Creating the App and Push Notification Service

Before going too much further we’re going to get our production app ready. We need an auth token for ZeroPush, which we can get for free from Heroku, since we’re going to deploy there anyway. If you don’t have a Heroku account already, follow their quickstart guide. Download their toolbelt or install the ruby gem, and login using their command-line tool.

Create your app on Heroku from the checked-out server code directory with heroku create. Before you can use add-ons on Heroku, you have to verify your account by adding a credit card number, but it will prompt you to do that when you create a ZeroPush service with heroku addons:add zeropush:inception. The inception tag refers to ZeroPush’s free plan. You shouldn’t get charged money for any of this unless you increase your pricing plan on Heroku or ZeroPush.

Once you’ve done that, we need the ZeroPush authorization tokens. You can get these either by going to Heroku’s website and following the ‘manage’ link, or running heroku config. There are two tokens, ZEROPUSH_DEV_TOKEN and ZEROPUSH_PROD_TOKEN. For now, we just need the dev token.

DotEnv and 12-Factor Development

If you’ve worked with Ruby in the last four years, you’re familiar with Bundler; if not, you’ll need to install it with sudo gem install bunder. Your Ruby installation may not need the sudo, but if it doesn’t it’s probably because you’re using rbenv or rvm and you probably already have bundler anyway.

Once you have bundler, install all of the app’s defined dependencies with bundle install.

The server’s Readme wants you to use the dotenv gem to do local development in the style of a 12-Factor App, where configuration is stored in UNIX environment variables. Copy the dotenv.sample file to .env and open it in your editor.

The dotenv gem isn’t in the Gemfile and the Sinatra app doesn’t use it, it just reads from the current environment. So you need to install the gem with sudo gem install dotenv. Or don’t: you can replicate its functionality with a single line of bash, which I’ll show you further down.

If you open it in your editor you’ll see some environment variables… notably there is DATABASE_URL that points to a PostgreSQL database, ZEROPUSH_DEV_TOKEN and ZEROPUSH_PROD_TOKEN for push notifications. Copy the values from heroku config into this file. To test things out you’re also going to need a PostgreSQL database at the address in the file, but since this post is about using Orchestrate I’ll leave that as an exercise to the reader. If you don’t want to do that exercise I’ll tell you which parts to skip.

You’ll also want to add AUTH_USER and AUTH_PASSWORD variables for HTTP basic auth, or else remove the use Rack::Auth::Basic block from config/init.rb. If you need an easy way to generate a strong password, I have this set to a function in my shell: ruby -r securerandom -e 'puts SecureRandom.urlsafe_base64(20, false)'.

Now, if you don’t want to install the dotenv ruby gem, you can simply add this line to the bottom of the .env script: bundle exec rackup, make the .env script executable (chmod +x .env), and be done with it; you can then start the app with ./.env; alternately if you did install the gem, you can start the app with dotenv bundle exec rackup. Skip this if you didn’t setup PostgreSQL. A schema will be created, and the app is ready for requests from the iOS client.

Setting up the Client

I’ve been creating web apps in Ruby for the past decade, so I blew through setting up the server in about ten minutes. I have done significantly less iPhone work, so this part took me a bit longer. Much of the instructions assume familiarity with the iPhone development process, and I hope to fill in the gaps.

Setup the Project and Build It.

Clone the PunchClock iPhone App.

Per the Readme, you’ll need to copy some files:

cp PunchClock/constants.h.sample PunchClock/constants.h
cp PunchClock/PunchClock_Beta.xcconfig.sample PunchClock/PunchClock_Beta.xcconfig
cp PunchClock/PunchClock_Debug.xcconfig.sample PunchClock/PunchClock_Debug.xcconfig

PunchClock uses CocoaPods to manage dependencies, and you can install that with sudo gem install cocoapods. Once you’ve got it, run pod install to download all of the “pods”.

You’ll need to edit PunchClock/constants.h to reflect your setup:

  • PCbaseURL: The first value will be your local dev server. This should be the “name” of your computer, suffixed with .local and the port number the sinatra app is running on, which defaults to 9292. You can find the name of the machine in System Preferences, Sharing Pane:


So, my value for the first PCbaseURL is dust.local:9292. For the second value, use the url of the app you created on Heroku.

  • backendUsername is the value you used for AUTH_USER in the .env file.
  • backendPassword is the value you used for AUTH_PASSWORD in the .env file.
  • keychainID tells the app where to store its user’s name in the iPhone’s keychain. Change the com.panic part to a domain that makes sense to you. I used io.orchestrate.
  • beaconUUID is, if you actually had an iBeacon device, the UUID of your iBeacon. If you don’t, it’s not a problem, but if you build the app and try to run it without a valid value here the app won’t load and you’ll get a debugger warning about Invalid parameter not satisfying: region != nil. So you have to make one up – a task for which I have another ruby one-liner in a shell function: ruby -r securerandom -e 'puts SecureRandom.uuid'.
  • geoFenceLat and geoFenceLong are the latitude and longitude coordinates of your office (or home office, in my case). Mine is a few miles away from Panic’s office in downtown Portland, Oregon, but if you don’t know yours, there are several free apps on the iOS app store that will tell you, if you search for “Geo Coordinates” you’ll find a suitable one.
  • geoFenceRadius is a distance in meters from the geoFenceLat and geoFenceLong coordinates which defines the boundaries of your office.

You can ignore the ‘hockey’ keys – they are for gathering issues from beta testers, and we’re not using that service.

  • zeroPushDevKey and zeroPushProdKey: set these to the values returned from heroku config earlier. They should already be in your .env file.

Finally, we can run our app in the iOS simulator. If you haven’t installed Xcode yet (you’ll need the latest of version 5, 6 might work too), install it now. Open the PunchClock.xcworkspace file if you haven’t already, which will open Xcode. Go to the device selector in the upper-left corner of the screen, to the right of ‘Punchclock’…


…and choose one of the iPhone options under iOS simulator.


Unless you’ve already setup Xcode with your iOS developer details, you won’t be able to install this to your phone yet. We’ll cover that in a bit – right now we just want to make sure everything works. Press the ‘Play’ button in the upper-left corner, it will build the app, launch iOS Simulator, and then launch the PunchClock app. The system will first prompt you to allow access to your location via the usual system dialog (hit allow), then the app will prompt you for your name. Since the simulator can’t do location with any accuracy (I believe it sets itself to Cupertino, Calif.), it will likely show you as ‘out’.


Get PunchClock on Your Phone

This section may be a bit of an aside – it’s not strictly necessary to do this in order to get the server talking to Orchestrate instead of PostgreSQL, but I don’t find the process intuitive and if you want to actually use the PunchClock app you’ll need to do this. So, we’re going to get the debug version running on your iPhone.

Join the iOS Dev Program

To proceed, you’ll need to be a member of Apple’s iOS Developer program, and it allows you to sign code such so you can run it on your iOS devices. Philosophical qualms about “freedom” aside, this does cost actual money – $99 per year. Once you’re a member, when you sign in to your Apple Developer Account from Xcode, (in Xcode’s Preferences, under “Accounts”), you should have a “signing identity” there:


If you go to Xcode’s “organizer” (Window -> Organizer) and plug your iPhone into your Mac, you should have a Provisioning Profile named “iOS Team Provisioning Profile"


And if you then select your phone from the device chooser (pictured above, when we deployed to the iOS simulator), you should be able to deploy to your phone now by pressing the play button. We’ll enable Push notifications later on in this post. First, we’re going to modify the server app to use Orchestrate instead of PostgreSQL.

Migrating PunchClockServer’s Storage Engine to Orchestrate

This is actually the easiest part of the whole process. Before I begin diving into the code, I like to take a few minutes to ask, what does this app look like? What are its requirements?

Examining the PunchClockServer App

PunchClockServer is a Sinatra app. If you’re not familiar with Sinatra, you might be familiar with its many clones#Frameworks_inspired_by_Sinatra), and I’d say that Sinatra probably had more of an impact on the web app ecosystem than Rails did – by making it dead simple to create small services. Open up the my_app.rb file, scroll down a bit, and you’ll see this:

get '/' do
  halt 404

This defines a handler for an HTTP method and route. When you GET ‘/’, it will respond with a 404. Sinatra apps are built from such declarations, and PunchClockServer has a handful:

  • GET /status: Renders the HTML template views/index.erb, which sets up a javascript listener to load /status/table once a minute and render it. This is used for something like Panic’s StatusBoard. It performs no data queries.
  • GET /status/table: Loads all of the people by name and status, the count of people who are “In”, and renders that in the HTML template views/table.erb. Used by the StatusBoard viewer from /status.
  • GET /status/list: Loads all the people and their statuses, and when given a person via the name query parameter, loads who they are ‘watching’ and who is watching them. It returns JSON, and is used by the iPhone app.
  • POST /status/update: This one’s a bit of a doozy. The phone uses this to update the web app with its status: “In” or “Out”. The PunchClockServer app loads the user from the name query parameter, updates their record, and sends push notifications to whoever is “watching” this user. It returns a simple status message via JSON.
  • POST /message/in: Loads all of the people whose statuses are “In” and sends them a push notification.
  • POST /watch/:target: Creates a “watching” relationship between the posting user (via the name query parameter) and the target user (via the target route parameter). If they are already watching the user, returns a special status message.
  • POST /unwatch/:target: Deletes the “watching” relationship between the posting user (via the name query parameter) and the target user (via the target route parameter).
  • GET /image/:name: Loads an image from the public folder. If one doesn’t exist for the requested user, loads unknown.png.

Modeling the Data

This is pretty straightforward:

We have a collection of “people”, and perhaps most importantly, this is focused on small teams. There is no authentication of the people, and they are identified by the name they gave when they signed up. You need to be able to list them, and filter and sort by certain parameters.

In Orchestrate, this can be a “people” collection. If we were going to be dealing with a large number of people, say, over a thousand, we’d very likely want a different way to query them, or perhaps organize them by their “team” into different collections, but a single collection will work for now.

There is a directional relationship between people we’ll call “watching”, whereby one person can be defined as watching another. Given a person, we can easily list all the people they’re watching via Orchestrate’s API. There isn’t a way to query the inverse, that is, who is watching a given person, unless we also create the inverse relation “watched by” alongside the “watching” relation.

Getting Your API Key

Sign up for an Orchestrate account if you haven’t already, and create an application from the dashboard, perhaps called “[your username]-PunchClock”. Get the API key for the application (it will display and select it when you click on it, copy that to your clipboard):


And add it to your .env file in the PunchClockServer app as ORCHESTRATE_TOKEN.

Setting up Dependencies

Now we need to install the appropriate ruby gems. Open your Gemfile file, remove the gem 'sinatra-sequel' and gem 'pg' lines. Where we’re going, we don’t need SQL.

I have some unreleased work on integrating Orchestrate’s graph relations types into the Ruby “object client“‘s interface (we’ll use this for People), and we’re going to make use of that, so until the next version of the orchestrate gem comes out, add this line with the git and branch declarations:

gem 'orchestrate', git: 'https://github.com/orchestrate-io/orchestrate-ruby', branch: 'mattly/61-relations'

Now, have bundler install the new dependencies with bundle install, and, watch as it goes down in flames:

[email protected]:~/c/o/PunchClockServer ±✦:master :) bundle install
Fetching gem metadata from https://rubygems.org/..........
Fetching additional metadata from https://rubygems.org/..
Resolving dependencies...
Bundler could not find compatible versions for gem "faraday":
  In Gemfile:
    orchestrate (>= 0) ruby depends on
      faraday (~> 0.9) ruby

    zero_push (>= 0) ruby depends on
      faraday (0.8.9)

Instead of guiding you through troubleshooting, I’ll just tell you that since PunchClockServer was released, a new version of the ZeroPush gem was released that resolves this problem. The Gemfile doesn’t specify the version to use, but the Gemfile.lock declares the “last known good version” at 2.3.0, released in March, and 2.4.1 as released on July 8th.

To resolve this, simply delete your Gemfile.lock and re-run bundle install.

Configuring the Server

Open up config/init.rb in your editor. Remove the require 'sinatra/sequel' line, and add require 'orchestrate' and require 'typhoeus/adapters/faraday'. Remove everything in the configure do and replace it such that it looks like this:

require 'sinatra'
require 'zero_push'
require 'orchestrate'

configure do
  DB = Orchestrate::Application.new(ENV['ORCHESTRATE_TOKEN']) do |faraday|
    faraday.response :logger
  PEOPLE = DB[:people]

We’re doing a few things in this block:

  1. Instantiating a new Orchestrate::Application client using our token defined in .env.
  2. Configuring the faraday adapter. We’re telling it to use a logger to STDOUT for responses, so we can watch the requests to Orchestrate.
  3. Assigning this client to the Constant DB.
  4. Assigning the “people” collection as an Orchestrate::Collection to the Constant PEOPLE.

Philosophical issues about globals, constants, classes being global instances of Class and whatnot aside, assigning a database to a constant is how you would generally interact with the Sequel gem when not using it’s ‘Model’ facilities. We’re planning an ActiveModel tie-in to help create model like People as a type of KeyValue object later. For now, we’re going to use the Orchestrate::KeyValue class from the recent 0.7 release.

There’s also one quick change to make in the my_app.rb file: add require 'time' somewhere among the other require statements at the top of the file.

Migrating the App’s Storage

Let’s proceed through the endpoints one by one:

GET /status/table

There are two calls to PostgreSQL here:

@people = Person.select_order_map([:name, :status])
@count = Person.where(:status => 'In').count

This assigns to @count the number of people currently “In”, and to @people an array of arrays, sorted by status; for my home office it would at the time of writing look like this:

[ ['mattly', 'In'], ['owlyssa', 'Out'] ]

Assuming we keep the data keys the same, it would be easy enough to replace that with:

@people = PEOPLE.sort_by {|p| p[:status] }.map{|p| [p[:name], [p[:status]]] }
@count = @people.select{|p| p[1] == 'In' }.count

Because Orchestrate::Collection uses the ruby Enumerable module to perform Orchestrate KeyValue LIST queries, it will fetch all of the people in the people collection, traversing pages if needed, and then perform Enumerable#sort_by and mapping as needed.

We already have the needed data for @count, so we don’t need to make a second trip to Orchestrate, we can use Enumerable#select to get our count.

GET /status/list

This endpoint returns JSON output, that for my home office would look like this:

[ { "status": "In", "name": "mattly", "watched_by_requestor": false, "watches_requestor": false },
  { "status": "Out", "name": "owlyssa", "watched_by_requestor": true, "watches_requestor": true } ]

It also updates the “status” of any person who hasn’t been updated in a day to “Stale”.

First up, we need a list of people. We’re going to replace this call to Postgres:

# people = Person.order(:name)
people = PEOPLE.to_a

We don’t need to worry about sorting the people, because we’re going to use people’s names as the keys in Orchestrate, and the LIST query returns KeyValue records sorted by key name. We can use Enumerable#to_a to turn the collections enumerator into a normal array.

If someone hasn’t checked in in 24 hours, we want to mark the as “stale”. That’s pretty straightforward:

stale = people.reject {|p| p[:status] == "Stale" }.
               select {|p| Time.parse(p[:date]) < Time.now - 24*3600 }

stale.each do |p|
  p[:status] = "Stale"

When we save Time objects to Orchestrate (as we will below), they’re transformed to a string representation by Time#as_json. If we want to use them as Time objects again, we’ll need Time.parse, which for some reason is in ruby’s extlib instead of stdlib. Time is part of stdlib by default, but Time.parse is not, and this is why we added the require 'time' declaration earlier.

If a ‘name’ query parameter is given in the request, we want to load that specific record. It’s a simple change:

  if params[:name]
    lowName = params[:name].downcase
    # requestor = Person[:name => lowName]
    requestor = people.find {|p| p[:name] == lowName }
    halt 400 unless requestor

We could also do PEOPLE[lowName] instead of sorting through the full list.

Next up, if there was a “requestor” indicated by the name query parameter, for each person, we need to know if they are watching that person, and if that person is watching the requestor. The app does it like so:

watched_by = p.watched_by_name(requestor.name)
watches = p.watches_name(requestor.name)

Which leverages some stuff in config/data.rb to do sequel joins. We’re going to replace that with Orchestrate’s Graph Type. The code for SQL actually makes two database calls per person (inefficiently, at that; it loads up every record instead of using SQL to filter), but we going to make only one:

watchings = people.inject({}) do |memo,p|
  memo.update({p.key => p.relations[:watching].to_a})

The p.relations bit is a branching point to the Graph types for Orchestrate, which is how you define relations between KeyValue objects. The :watching bit indicates the “type” of relation, and using Enumerable#to_a simply tells the client that we want the results as an array.

We’re using Enumerable#inject on the people array to construct a Hash object (Pythonistas will know this as a Dict), and Functional programmers will more likely know #inject as fold.

Now, we’re going to modify the iteration over people to output the results we want. The old code is commented out and the new code is directly below that.

# output = []
# people.each do |p|
output = people.map do |p|
  # watched_by = false
  # watches = false
  out = { status: p[:status],
          name: p[:name],
          watched_by_requestor: false,
          watches_requestor: false }

  if requestor
    out[:watches_requestor] = watchings[p.key].include?(requestor)
    out[:watched_by_requestor] = watchings[requestor.key].include?(p)
    # watched_by = p.watched_by_name(requestor.name)
    # watches = p.watches_name(requestor.name)

  # output << { "status" => p.status, "name" => p.name, "watched_by_requestor" => watched_by, "watches_requestor" => watches }

We’re doing three things differently here:

  1. Using #map instead of #each and a separate variable to accumulate our output.
  2. Replacing the intermediate watched_by and watches variables with their place on the data to be outputted.
  3. Using Enumberable#include? on the results of our Graph GET query to determine if any given person is watching the requestor, or if the requestor is watching any given person.

And now the phone app can get its list from Orchestrate instead of SQL. Next up, the phone will be able to update things.

POST /status/update

This is a big one. I ended up replacing the entire DB.transaction block for doing an SQL transaction. Here’s a walkthrough:

# person = Person.for_update.first(:name => lowName)
person = PEOPLE[lowName]

We’re using people’s name as the key, so retrieving them is easy.

if ! person
  # person = Person.new()
  person = Orchestrate::KeyValue.new(PEOPLE, lowName)
  person.value = {'name' => name}

This bit is actually going to help improve the Ruby client – I realized that having a way to instantiate a ‘stub’ for a new KeyValue from a collection that hasn’t been saved yet would be useful, so you don’t have to do it yourself like I did there.

#recipient_ids = []
#person.watchers.each do |w|
#  if w.push_id != ""
#    recipient_ids << w.push_id
#    puts "Queuing notification for #{w.name}"
#  end
people = PEOPLE.to_a
watchings = people.inject({}) do |memo,p|
  memo.update({p.key => p.relations[:watching]})
recipient_ids = people.select{|p| watchings[p.key].include?(person) }.
  each{|p| puts "Queuing notification for #{p[:name]}"}.
  map{|p| p[:push_id] }

Get the list of people, find out who everyone is watching, filter it down to just those watching our person who recently changed their status, and get their push_id so we can send them a notification.

Perhaps at this point I wonder if creating a reversed directed relation watched_by would be useful, so I don’t have to iterate over everybody whenever I want to know who is watching someone. It’s given me some ideas for the eventual ActiveModel tie-ins.

# person.status = params[:status]
# person.name = lowName
# person.push_id = params[:push_id]
# person.beacon_minor = params[:beacon_minor]
# person.version = agent_version
# person.date = DateTime.now
# person.save or {"result" => STATUS_ERROR, "reason" => "The record could not be saved"}.to_json
  name: params[:name],
  status: params[:status],
  push_id: params[:push_id],
  beacon_minor: params[:beacon_minor],
  version: agent_version,
  date: Time.now
if ! person.save
  output = {result: STATUS_ERROR, reason: "Record could not be saved"}

The main difference here is, since there aren’t method accessors on the KeyValue object for properties, we’re updating them with Hash#update. Note that the original code used a transaction to ensure that the row loaded from PostgreSQL couldn’t be modified until it was updated – it locks the row pessimistically. In Orchestrate, and with Orchestrate::KeyValue in particular, this is done optimistically, with the Ref value for the key value object.

Normally with Orchestrate, you have to manage this yourself by providing an If-Match header with your request. Orchestrate::KeyValue will do this automatically for you. And, in the case of the user who just signed up because their record didn’t exist yet (and therefore has a ref of nil), it sends an If-None-Match header to ensure that the save only happens of the user with that key doesn’t exist yet. All of that happens automatically with Orchestrate::KeyValue#save.