If you've ever worked with a relational database, then you're probably already familiar with the concept of field filtering. Instead of retrieving every field in a complex record, you select only the fields you specifically need for the task at hand. For example, instead of this SQL query:


...you'd probably write something more like this:

SELECT id, first_name, last_name FROM USERS

It's a simple concept, but incredibly convenient and efficient, and today I'm delighted to announce the availability of JSON Field Filtering operations across the entire Orchestrate API.

Design of JSON Field Filtering

We've given a lot of thought to this feature. JSON fields represent a hierarchy of information with much more potential for complexity than the tabular results of a relational database. But we've come up with an elegant design that supports a wide variety of use cases, and we think you're going to love it.

First of all, field filtering is available everywhere in the API where JSON results are served: from simple HTTP GET requests to key-ordered listings and search queries, across all of our data types: key-value items, time-series events, and graph relationships.

By using the new with_fields and without_fields querystring parameters, users can whitelist and/or blacklist certain fields according to their fully-qualified names.

For example, take a look at this JSON record:

  "a" : 1,
  "b" : [{
    "c" : 1,
    "d" : 2
    "c" : 3,
    "d" : 4

An application dependent upon this data might only need to retrieve the value of the "a" field in some cases, and we could omit the value of the "b" field entirely. It should come as no surprise then that are two different ways to filter the fields:

curl -XGET http://api.orchestrate.io/v0/my_collection/my_key

curl -XGET http://api.orchestrate.io/v0/my_collection/my_key

With either query the resultant filtered JSON would look like this:

{ "a" : 1 }

But wait! It gets better!

We can actually use both the with_fields and without_fields predicates at the same time to achieve more advanced filtering scenarios.

The important thing to understand here is that the with_fields predicate is applied first, causing all the designated field values to be cloned into a new object. If the with_fields predicate is empty, we copy all the fields by default. Then, we apply the without_fields predicate to the resultant JSON and subtract all the designated field values from the cloned subset.

Just remember: Whitelisting comes before blacklisting.

When used in concert, the two operations are most useful when blacklisting a subordinate field of an object whose values we've already whitelisted. So, given our previous example, we could combine a with_fields=value.b clause with a without_fields=value.b.d to create the following result:

  "b" : [{
    "c" : 1
    "c" : 3

By designating the deeply-nested field name value.b.d as a blacklist field, we erase that value from all the deeply-nested children, even within arrays. You can use this functionality to optimize payload sizes, or to avoid retrieving sensitive data when it's not absolutely necessary.

The optimization aspects of this feature are especially convenient in our Search API, or when paginating through Event listings, when it's commonplace to iterate over hundreds or thousands of records. In those cases, retrieving only the exact subset of data you need helps keep your application lean and fast.

Partial Updates

Field filtering also combines nicely with our JSON Patch API, which lets you execute partial updates on stored JSON objects, updating individual fields without having to re-submit the entire record.

Now, with field filtering of response JSON, you can effectively query for a single field and then update that same individual field, without having to ever interact with the rest of the record. For example, to query for a user's current phone number, and then subsequently update that field, you might do something like this:

curl -XGET http://api.orchestrate.io/v0/user/ABC123

...which would give you a result like this, even if the complete user JSON object stored in Orchestrate has hundreds of different fields:

  "user_id" : "ABC123",
  "phone_number" : "617-555-1212"

With that information, you could prompt the user for an updated phone number and then issue a PATCH operation to execute the update:

curl -i "https://api.orchestrate.io/v0/$collection/$key"
  -H "Content-Type: application/json-patch+json"
  -d '[{ "op": "replace", "path": "value.phone_number", "value": "503-867-5309" }]'

Your Platform in Motion

As the Orchestrate API continues to grow and evolve, it's nice to see how these kinds of features fit together like puzzle pieces, enabling more and more complex use cases and helping our customers build more sophisticated applications.

The field filtering API is available immediately on all our data centers and will be integrated soon into our client libraries. We can't wait to see what you'll build next!

Remember: Our lines of code, our technology -- is your platform. We’ll never stop improving and iterating new products and services to better your business.