Let's say you're looking for an authentication module. Your search on Scout JS returns the best modules, based upon a number of factors:
- Recent downloads of the module
- Stars on the GitHub repo
- Forks of the GitHub repo
- How recently the repo has been updated
To get all of these data sources in one place we turned to our AppFog Platform-as-a-Service, NoSQL Database-as-a-Service Orchestrate, and the multi-talented Asa Miller at Tiny Creative. The result, as you might expect, is a Node application, itself using 37 modules. We'll get into the details in future posts, but feel free to browse the repositories.
Scout JS is more than a developer tool; it's a sample application. It covers some common patterns, such as combining disparate sources, denormalizing NoSQL data, building an API server, and running non-web AppFog instances.
Combine Data Sources
The base of Scout JS's dataset are the npm's modules, which are inching up toward 200,000. Orchestrate actually has a tutorial to replicate npm, which includes tapping into its changes feed.
The npm data, which includes the download history, needs to be updated on an ongoing basis, which is stored separately from the GitHub data.
Similarly, GitHub needs to be updated frequently, because developers are starring and forking repositories as you read this post. Working within the GitHub API limits, Scout JS grabs the latest data and stores it in its own collection.
Asa shared a step-by-step tutorial combining GitHub and npm data.
Denormalize NoSQL Data
Database normalization is more commonly discussed with relational databases. In SQL databases you separate your tables, then use joins to bring data together based on identifiers. Scout JS borrows a page from relational databases, but then brings everything together a single collection for searching.
You can use concepts from this NoSQL denormalization post, which discusses an approach that continually looks for changes in your own database. For this project, we know every time data changes. So we can run denormalization whenever underlying GitHub or npm data for a module changes.
Asa's tutorial includes denormalization code.
Build an API Server
Scout JS runs on Hapi, a lightweight Node framework. That Hapi keeps things simple is exactly why it's great for an API server. You can simply create routes to handle the types of requests you expect from your application.
Run a Non-Web AppFog Instance
Not everything you build is a website or even web application necessarily. For example, you might have an API server like above, or even something that just chugs along calling out to other APIs. In the second case, you'll want a standalone application.
You'll want to
cf push with the
--no-route flag, as discussed in these Cloud Foundry docs (AppFog is based on Cloud Foundry). Also note you still need to listen on the port that Cloud Foundry specifies, so it can perform its health checks.
Organize the Frontend with React and Redux
In addition to those great backend insights, Scout JS is an example of a slick React-based user interface. React is combined with a state-container called Redux that has recently become popular. Redux is another take on Facebook's Flux. In fact, Redux's popularity shows in its Scout JS chart (above), as well as the fact that it's the second result when searching "flux."
Of Course It's Open Source
Open Source is important to CenturyLink, and we work with a number of open source communities, including Docker, Chef, Elasticsearch and Cloud Foundry projects. CenturyLink has made significant contributions to Panamax, Lorry, Image Layers, Chef integration with VMware, Iron Foundry, Cloud Foundry, xUnit.net, Chocolatey, and ElasticLinq. And that's just in the last year. So, providing a tool to the Node community was a natural progression, as was supporting Asa in open sourcing the code.
The code behind Scout JS also does double duty as a great example of an application running on our Cloud Foundry-based AppFog. If you're ready to dive into your next project, using Node or another runtime, you can easily get started with AppFog and leave the infrastructure details to us.