There are about 58 billion tweets – and over 2 billion Twitter search queries – every single day. If you want to track specific topics on Twitter, you can do what millions of other folks have done – and build a Twitter bot. In this tutorial, I’ll walk you step by step through the process of building a Twitter bot using Node.js, Orchestrate and Twit.

Since I’m especially interested in gaming, I decided to build a gaming twitter bot to aggregate all of the tweets with “#gaming #games, #xbox, etc…”. It sounds like gamedev, but we are searching for all tweets related to gaming, not just development tweets. Of course, you can customize your own bot to track a subject of your choice.

Prepare the Framework

The skeleton of my app will be Express, which you can install here. You can see the final project’s source code here. Aside from Express, our bot primarily leverages the Twit and orchestrate.js modules.

Orchestrate will make leveraging Twitter data much easier with its built-in Search queries. First, let’s get our app.js file setup. Out of the box, you will see something like the following:

var express = require(‘express’);
var router = express.Router();

/*Get home page */
router.get(‘/‘, function (req, res) {
  res.render(‘index’);
});

modules.exports = router;

This gives us the starting point we need. First, let’s include the modules we need:

var db = require('orchestrate')(process.env.ORCHESTRATE_API_KEY)
  , Twit = require('twit’)

That process.env.ORCHESTRATE_API_KEY refers to the environment variable where we store our Orchestrate credentials. By putting it in an environment variable, we keep things like passwords out of our code. If you’re using Mac OS X or any Linux distribution, you can set environment variables like this:

export ORCHESTRATE_API_KEY=YOUR_API_KEY

When you deploy your app, remember to set your environment variables wherever you deploy the app. For example, on Heroku, check out Configuration and Config Vars for how to do that.

Stream the Hashtags

Next up, Twit. I chose to use the Twit module was because it seemed far more developed and user friendly to make getting the data we want to compare and manipulate much easier.

So next we need to make sure we load up our credentials from Twitter. You’ll need to set TWITTER_API_KEY, TWITTER_API_SECRET, TWITTER_ACCESS_TOKEN, and TWITTER_ACCESS_TOKEN_SECRET as environment variables. To get those credentials from Twitter, follow these steps:

  • Create an application
  • Click “manage API Keys”
  • API key is your TWITTER_API_KEY
  • API secret is your TWITTER_API_SECRET
  • Click “Create my access tokens”
  • Watch this seal learning to swim while you wait for your access tokens to generate
  • Refresh the API keys page; your access tokens should appear
  • Access token is your TWITTER_ACCESS_TOKEN
  • Access token secret is your TWITTER_ACCESS_TOKEN_SECRET

Once you have the above keys fill this chunk of code out, set them as environment variables just like we did for ORCHESTRATE_API_KEY. Then, invoke Twit like this:

var T = new Twit({
    consumer_key:         process.env.TWITTER_API_KEY
  , consumer_secret:      process.env.TWITTER_API_SECRET
  , access_token:         process.env.TWITTER_ACCESS_TOKEN
  , access_token_secret:  process.env.TWITTER_ACCESS_TOKEN_SECRET
});

Great! Now we have initialized all of the modules we are going to use. Now we need to tell twitter what tweets we want. Which looks like this:

var stream = T.stream('statuses/filter', {
  track: '#gaming, #videogames, #games, #gameclash, #gamedev, #xbox, #xboxlive, #xbox360, #xboxone, #xboxlive, #cod',
  language: 'en'
});

The track parameter tells Twit which topics to follow, and thus, which tweets to return. With the settings above, it will grab any tweet in English with words like gaming and gamedev. Twit does the rest. So now it is grabbing those tweets, but how do we see them? Simple, we create a listener. We are going to use the stream variable that we just created.

To grab each tweet as it comes in you we will do this:

stream.on('tweet', function (tweet) {
 console.log(tweet);
});

If you were to restart your node.js app you would be able to see in your terminal every single tweet as it comes in. But just logging these tweets does nothing for us! We love data and want to be able to use it to our advantage.

Store the Tweets

Enter Orchestrate. Orchestrate makes it so easy. Are you ready for this?

We are going to store each tweet by its ID, create a collection in Orchestrate called tweets (or a name of your choice), and store the entire tweet object.

/** the stream **/
stream.on('tweet', function (tweet) {
 db.put("tweets", tweet.id, tweet)
 .fail(function (err) {
  console.error(err);
 });
});

We replaced the console.log with a db.put and a few extra parameters and we are done. We use console.error to log any failures to add tweets to Orchestrate. If you want, you can replace console.error with your own error handling to proactively handle any unexpected failures.

Now we add the routes our frontend will consume: searching and displaying tweets.

We are going to use the db.searchNewBuilder() functionality from the node.js module. I want my display to start with ten, I want to search all of them, and display. Here is what the code looks like:

/* GET home page. */
router.get('/', function(req, res) {

 db.newSearchBuilder()
   .collection("tweets")
   .limit(10)
   .offset(0)
   .query("*")
   .then(function (results){
     if(results.body.count > 0){
       res.render("index", {tweets: results.body.results});
     }
   });
});

Now you are probably thinking, “Ok so it’s that simple, and easy, but how do I get the other tweets that I have stored?” Well the answer to that is just as easy with Pagination, and using this same chunk of code with some minor tweaks.

I made a new route in express with the path of /load/tweets/:page. You can make the path whatever, just be sure to add that :page bit, AKA a path variable to pass the page number. Now our route can return successive pages of tweets:

router.get('/load/tweets/:page', function (req, res){
   db.newSearchBuilder()
     .collection("tweets")
     .limit(10)
     .offset((parseInt(req.params.page) * 10) - 10)
     .query("*")
     .then(function (results){
       console.log(results.body.count);
       res.json({results: results.body});
     })
     .fail(function (err){
       console.log(err);
     });
});

I want to clarify what I wrote for offset: the next page number * 10 (original limit) – 10 (last set to be sent) = new offset.

That’s it for the server. Now you can start your server by running node index.js, which will start the app on http://localhost:3000. You can now explore all the routes we just wrote. But, they’re all in JSON, rather than a pretty graphical interface. I leave building that as an exercise for the reader — or you can install the tweetbot and see an example interface for yourself, complete with infinite scrolling.

Frontend in hand, you now have a functional tweetbot that will consolidate all of the tweets for gaming, or whatever topics you want to follow. Enjoy!

(Image courtesy of Marek Sotak)