Node developers love modules, which allow them to separate functionality and build on top of the open source efforts of others. Modules are downloaded billions of times every month, with npm serving as the nervous system for the server-side JavaScript community. Signals can sometimes get crossed when it comes to finding the best modules. That's why we helped create Scout JS to take a more holistic view of a module's place within the community, merging npm's own data with GitHub. And true to CenturyLink's support of open source, all of the code is available.

Search authentication on Scout JS

Let's say you're looking for an authentication module. Your search on Scout JS returns the best modules, based upon a number of factors:

  • Recent downloads of the module
  • Stars on the GitHub repo
  • Forks of the GitHub repo
  • How recently the repo has been updated

To get all of these data sources in one place we turned to our AppFog Platform-as-a-Service, NoSQL Database-as-a-Service Orchestrate, and the multi-talented Asa Miller at Tiny Creative. The result, as you might expect, is a Node application, itself using 37 modules. We'll get into the details in future posts, but feel free to browse the repositories.

Scout JS is more than a developer tool; it's a sample application. It covers some common patterns, such as combining disparate sources, denormalizing NoSQL data, building an API server, and running non-web AppFog instances.

Combine Data Sources

The base of Scout JS's dataset are the npm's modules, which are inching up toward 200,000. Orchestrate actually has a tutorial to replicate npm, which includes tapping into its changes feed.

The npm data, which includes the download history, needs to be updated on an ongoing basis, which is stored separately from the GitHub data.

Similarly, GitHub needs to be updated frequently, because developers are starring and forking repositories as you read this post. Working within the GitHub API limits, Scout JS grabs the latest data and stores it in its own collection.

Asa shared a step-by-step tutorial combining GitHub and npm data.

Denormalize NoSQL Data

Database normalization is more commonly discussed with relational databases. In SQL databases you separate your tables, then use joins to bring data together based on identifiers. Scout JS borrows a page from relational databases, but then brings everything together a single collection for searching.

You can use concepts from this NoSQL denormalization post, which discusses an approach that continually looks for changes in your own database. For this project, we know every time data changes. So we can run denormalization whenever underlying GitHub or npm data for a module changes.

Asa's tutorial includes denormalization code.

Build an API Server

Scout JS runs on Hapi, a lightweight Node framework. That Hapi keeps things simple is exactly why it's great for an API server. You can simply create routes to handle the types of requests you expect from your application.

Run a Non-Web AppFog Instance

Not everything you build is a website or even web application necessarily. For example, you might have an API server like above, or even something that just chugs along calling out to other APIs. In the second case, you'll want a standalone application.

You'll want to cf push with the --no-route flag, as discussed in these Cloud Foundry docs (AppFog is based on Cloud Foundry). Also note you still need to listen on the port that Cloud Foundry specifies, so it can perform its health checks.

Organize the Frontend with React and Redux

Scout JS search for flux shows redux

In addition to those great backend insights, Scout JS is an example of a slick React-based user interface. React is combined with a state-container called Redux that has recently become popular. Redux is another take on Facebook's Flux. In fact, Redux's popularity shows in its Scout JS chart (above), as well as the fact that it's the second result when searching "flux."

Of Course It's Open Source

Open Source is important to CenturyLink, and we work with a number of open source communities, including Docker, Chef, Elasticsearch and Cloud Foundry projects. CenturyLink has made significant contributions to Panamax, Lorry, Image Layers, Chef integration with VMware, Iron Foundry, Cloud Foundry,, Chocolatey, and ElasticLinq. And that's just in the last year. So, providing a tool to the Node community was a natural progression, as was supporting Asa in open sourcing the code.

The code behind Scout JS also does double duty as a great example of an application running on our Cloud Foundry-based AppFog. If you're ready to dive into your next project, using Node or another runtime, you can easily get started with AppFog and leave the infrastructure details to us.