Marvel recently opened up an API for its character and comic data, which includes all of the characters, what comics they appear in, and a lot more. After poking around with it for a little while, I decided I wanted a more visual way to explore the characters and the comics, and that this would be a great project to build off of MongoDB and CenturyLink Cloud. We could use graph search to explore the relationship between characters and comics, as well as full text search to explore the data.

This project was pretty fast to build. It took a little time to import the data from the Marvel API and to do the design and frontend work, but the actual backend code and database work was really fast (less than a day or so).

What we’re building

A node.js app for searching and viewing all the Marvel Comics data. Since Marvel’s API is rate limited, we’ll store all the data in MongoDB and periodically update it from the API. It took some time to get the data from Marvel’s API and I won’t go over that part here, but if you want to download the cached data, I’ve included a dump of it in the repo.

View the code on GitHub.

View the finished app here.

explore Marvel Comics

Wolverine

The Tech

Deploy a New Virtual Server with MongoDB

If you don't have a CenturyLink Cloud account yet, head over to our website and sign up for a free trial. You'll need it to access CenturyLink Cloud products.

Our first step is to deploy a new CenturyLink Cloud virtual server. Follow the steps below.

  1. Log into the CenturyLink Cloud control portal at https://control.ctl.io/
  2. On the left side menu, click Infrastructure and then Servers.

    Marvel

  3. On the left-hand side of the server panel, click on the region for the server we will provision.

  4. Click create and then server.
  5. Complete the setup form for your new server. Be sure to fill out the fields for server name and admin/root password.
  6. For operating system, select "CentOS 7 | 64-bit".
  7. Click create server.
  8. Your server provisioning request will enter the queue. You can watch the progress of your request on the screen. Your server is provisioned when the status of all tasks in the queue is complete.

    Marvel

  9. After your new server is provisioned, in the CenturyLink control portal, click Infrastructure on the left side menu, and then click Servers.

  10. Navigate to your new server and click on its name.
  11. Click the more menu, and then click add public ip.
  12. Check the box for SSH/SFTP (22).
  13. Click custom port... and then single port.
  14. Type "27017" in the blank box to open up the MongoDB server port.

    Marvel

  15. Click add public ip address.

Installing and Configuring MongoDB

  1. Navigate to your server in the CenturyLink Cloud control panel as in the previous section. Your server's public IP address will be noted on the screen.
  2. From a shell on your local machine, connect to your new server with the following command. Replace "YOUR.VPS.IP" with your server's public IP address.
    ssh [email protected]
    
  3. Install the MongoDB server software by running the following commands.
    $ yum install -y mongodb mongodb-server
    
  4. With your favorite text editor, open /etc/mongod.conf. Look for the line that begins "bind_ip" and comment it out. The top of your file should now look like this:

    ##
    ### Basic Defaults
    ##
    
    # Comma separated list of ip addresses to listen on (all local ips by default)
    #bind_ip = 127.0.0.1
    
  5. Start the MongoDB service by running the following command.
    $ service mongod start
    

Starting Off the Application

First we’ll start off with a basic Express app. We’ll create a web.js file for our express routes and a functions.js file for our (yup you guessed it) app functions.

In the web.js we’ll start out with this:

var express = require('express');
var exphbs  = require('express-handlebars');
var querystring = require('querystring');

// Stuff for Express
var logger = require('morgan');

// our functions
var f = require('./functions.js');

var app = express();
app.use(logger('dev'));
app.use(express.static(__dirname + '/public'));

// HANDLEBARS HELPERS
var hbs = exphbs.create({ defaultLayout: 'main' });
app.engine('handlebars', hbs.engine);
app.set('view engine', 'handlebars');

var helpers = require('./handlebars-helpers.js')(hbs.handlebars);

//  ROOT SEARCH INPUT PAGES
app.get('/', function(request, response) {
  response.render('search-characters');
});

//  404 page
app.get('*', function(request, response) {
  response.render('error');
});

// START THE SERVER
var port = process.env.PORT || 5000;
app.listen(port, function() {
  console.log('Listening on ' + port);
});

Viewing a character

Let’s start off with viewing a specific character’s page. We’ll use the character id provided from the Marvel API as our key.

Add this to your web.js file so we can respond to /character/123456 requests.

app.get('/characters', function(request, response) {
  response.render('search-characters');
});

// GET A SINGLE CHARACTER
app.get('/character/:id', function(request, response) {
  var id = request.params.id;

  var page = parseInt(request.query.p, 10) || 1;
  var limit = 20;
  var offset = (page - 1) * limit;

  f.getCharacter(id, offset, limit)
    .then(function (data) {
      // console.log(data);
      // check if we have data, otherwise show the no data screen
      data.hasData = (data.wiki || data.description) ? true : false;

      data.linkback = 'http://marvel.com';
      if (data.urls) data.linkback = data.urls[0].url;

      var pages = Math.ceil(data.comics.total / limit);
      response.render('character', {
    pagination: {
      page: page,
      pageCount: pages,
      needed: (pages > 1)
    },
    data: data
      });
    })
    .fail(function(error){
      console.log(error);
      response.render('error');
    });
});

Then in the functions.js file, create a new function that will get a character’s data from MongoDB.

// GET A SINGLE CHARACTER
// passed in the character ID and optional pagination page
exports.getCharacter = function (charId, page) {
  var out = {};
  var comicResults;
  var deferred = Q.defer();

  // Normalize stuff.
  charId = parseInt(charId);
  page = parseInt(page) || 1;

  MongoClient.connect(mongodbUrl, function (err, db) {
    var collection = db.collection('characters');

    collection.findOne({'id' : charId})
      .then(function (character) {
        console.log("FOUND", character.name);
        out = character;

        return getComicsByCharacter(db, character, (page - 1) * 20)
      })
      .then(function (comics) {
        cleanUpCharacterData(out);
        out.comics = comics;

        deferred.resolve(out);

        db.close()
      })
      .catch(function (error) {
        console.log("ERROR", error);
      });
  });

  return deferred.promise;
}

Getting data from MongoDB is super easy. This will return the data stored at the key we pass in. But we want to get more than just the character data. We also want to see the comics that the character appears in. You can see the getComicsByCharacter call.

We’ll create the function that gets the comics.

// GET COMICS BY CHARACTER ID
// passed in the MongoDB database, Character object and optional pagination page
function getComicsByCharacter (db, character, offset, limit) {
  var deferred = Q.defer();
  var comicIds = [];
  offset = offset || 0;
  limit = limit || 20;

  // Collect comic IDs
  character.comics.items.forEach(function (item) {
    if (0 < item.id) {
      comicIds.push(item.id);
    }
  });

  if (0 < comicIds.length) {
    // Search MongoDB comics collection
    db.collection('comics').find({'id': {'$in': comicIds}})
      .sort({'id': 1})
      .skip(offset)
      .toArray(function (err, docs) {
        if (null != err) {
          deferred.reject(err);
        }
        else {
          deferred.resolve(docs.slice(0, limit));
        }
      });
  }
  else {
    deferred.resolve([]);
  };

  return deferred.promise;
}

Iron Man

Text Search

Full Text Search

Now we can view a character, but how about searching? MongoDB supports full text search as of version 2.4. This allows us to do a full search across any text-based field we decide to index. To use this powerful feature, connect to the database server with this command from your server shell.

mongo localhost/marvel

Then, from the mongo client prompt, run these commands:

db.characters.ensureIndex({
  'name': 'text',
  'wiki.real_name': 'text',
  'wiki.alias': 'text',
  'description': 'text',
  'wiki.occupation': 'text',
  'wiki.place_of_birth': 'text',
  'wiki.groups': 'text',
  'wiki.relatives': 'text',
  'wiki.hair': 'text',
  'wiki.powers': 'text',
  'wiki.abilities': 'text'
}, 'charactersTextIndex');

db.comics.ensureIndex({
  'title': 'text',
  'issueNumber': 'text',
  'description': 'text',
  'creators': 'text'
}, 'comicsTextIndex');

Now let's add full-text search to the application. First create a new endpoint for the search in the web.js file.

app.get('/characters/search', function(request, response) {
  var query = request.query.s;
  var field = request.query.field || '';
  var gender = request.query.gender || '';
  var reality = request.query.reality || '';

  var page = parseInt(request.query.p, 10) || 1;
  var limit = 20;

  var offset = (page - 1) * limit;

  f.getCharacters({
    offset: offset,
    query: query,
    limit: limit,
    gender: gender,
    field: field,
    reality: reality
  })
    .then(function (data) {
      var pages = Math.ceil(data.length / limit);
      data = data.slice(0, limit);

      // kill the page object before we serialize the qs
      delete request.query.p;

      response.render('characters', {
    pagination: {
      page: page,
      pageCount: pages,
      qs: querystring.stringify(request.query)
    },
    data: data
      });
    })
    .fail(function(error){
      console.log(error);
      response.render('error');
    });
});

Then add a new function getCharacters() to functions.js:

// SEARCH CHARACTERS
// This converts query parameters into a MongoDB search.
exports.getCharacters = function (options) {
  var options = options || {};
  var builtSearch = {};
  var deferred = Q.defer();
  var defaultSort = { 'name' : 1 };
  var projectFields = null;

  // Normalize options.
  options.offset = parseInt(options.offset) || 0;

  if (options.field) {
    // For a real name, we search both 'wiki.real_name' and 'wiki.alias'
    if (options.field == 'wiki.real_name') {
      builtSearch['$or'] = [
        { 'wiki.real_name' : { '$regex' : options.query,
                               '$options' : 'i' } },
        { 'wiki.alias'     : { '$regex' : options.query,
                               '$options' : 'i' } }
      ];
    }
    else {
      builtSearch[options.field] = {
        '$regex' : options.query,
        '$options' : 'i'
      };
    }
  } else {
    // This builds a MongoDB full text search using the text index.
    builtSearch['$text'] = { '$search' : options.query };
    projectFields = { 'score': {'$meta': "textScore"} };
    defaultSort = {'score': {'$meta':"textScore"}};
  }

  if (options.gender) {
    builtSearch['gender'] = options.gender;
  }

  if (options.reality) {
    builtSearch['wiki.universe'] = options.reality;
  }

  console.log("SEARCH characters", JSON.stringify(builtSearch));

  return searchCollection ('characters', builtSearch, options.offset, defaultSort, projectFields);
}

function searchCollection (collectionName, query, offset, sortCriteria, project) {
  var deferred = Q.defer();
  offset = offset || 0;

  // Execute the MongoDB search
  MongoClient.connect(mongodbUrl, function (err, db) {
    var collection = db.collection(collectionName);

    var cursor = collection.find(query);

    if (null != project) {
      cursor = cursor.project(project);
    }

    cursor.sort(sortCriteria)
      .skip(offset)
      .toArray(function (err, docs) {
        if (null != err) {
          deferred.reject(err);
        }
        else {
          console.log("SEARCH found", docs.length);

          deferred.resolve(docs);
        }

        db.close();
      });
  });

  return deferred.promise;
}

Marvel Explore

Back to our query, our search form will pass a few parameters. The getCharacters function is already set up to process different parameters using special MongoDB keywords like $regex.

Marvel Search

Comics

The process to get a comic or search through them is very similar. Take a look at the source code on GitHub to learn more about searching for comics.

Next steps

Now our app can view and search. We use full-text search to find comics and characters, but what else could we do? What about seeing which characters appear in comics together, or seeing which villains have appeared in the most comics? There’s a lot more we could do with this data. Enjoy!