Set up Node.JS, MongoDB, and Express on Free Spatial Web Hosting - Archived

After I finished my post on how to wire up MongoDB + Python/Flask + OpenShift, I felt bad for all the Node.JS developers that weren’t getting any love. So in attempt to be more polyglotie I ported my flask web service to Node.JS using the Express framework. If you are a Python developer and wanted to learn more about Node.JS here is your chance to see a small web service ported over. If you read the last post you will notice there is some repition as I explain the concepts again to Node.JS developers.

MongoDB Spatial with Node.JS and Express on OpenShift

After I finished my post on how to wire up MongoDB + Python/Flask + OpenShift, I felt bad for all the Node.JS developers that weren’t getting any love. So in attempt to be more polyglotie I ported my flask web service to Node.JS using the Express framework. If you are a Python developer and wanted to learn more about Node.JS here is your chance to see a small web service ported over. If you read the last post you will notice there is some repition as I explain the concepts again to Node.JS developers. The only prior knowledge you need is to have read my original post about getting spatial going in MongoDB on OpenShift. There are several goals for this article:

  • Learn a little bit about Express – a Node.JS web framework
  • Learn about how to connect to MongoDB from Node.JS
  • Create a REST Style web service to use in our SoLoMo application

I hope by the end you can see how using a Platform as a Service can get you going with Node.JS, MongoDB, and Spatial faster than you can say…“Awesome Sauce”. We have a lot of ground to cover so let’s dig right in.

Why Node.JS and why the Express framework?

Since Node.JS is one the newer kids on the block, I am going to do a very short introduction into some of the reason people use Node.JS for their web applications.

Node.JS is a web server and code execution environment for running JavaScript. Rather than just using JavaScript as code for your web pages, you can also use JavaScript to write your server-side applications. There ae several areas in which Node.JS shines for web application development:

  • Low response time or high client concurrency server code
  • Applications that want to maintain connections between client and the server
  • Teams that want to standardize on using one programming language for client and server

There are great pages here, here, here, and here that cover some of the strengths and weaknesses of using Node. Like any tool – it is right for some situations and not for others.

There are also quite a few great resources online for getting started with programming in Node, including this, this, different node books, and of course, the documentation on the Node.JS site itself.

A popular addition to Node.JS is the Express framework. As a web framework Express was heavily modeled off of Ruby’s Sinatra framework, which also makes it similar to the Flask framework in Python. As a web framework, Express bring features you commonly use in writing server-side web code, such a templating and URL binding. Today we are only going to use the URL binding to build some REST style web services. Here are some basic introductions to Express, including the one on the project site.

Creating the Node.JS application

Let’s get started on our application. Here is OpenShift the command line to create the Node.JS app and add a mongo cartridge to the application

rhc app create nodews nodejs-0.6 
rhc cartridge add mongodb --app nodews

By default the main file that starts your application is server.js (this is pretty standard among Node.JS hosting services).

Modifying the Express template

I have created a simple template app in github to start our coding. It has Express, a MongoDB example, and some simple URLs defined. Please go ahead and merge this github example into your code:

cd nodews
git remote add upstream -m master git://github.com/openshift/simple_node_express_mongo.git
git pull -s recursive -X theirs upstream master

Now we have a template that can we modify to make our spatial app.

The setup section of the template is where all the parameters of the app are defined and initialized. But the primary way the app works is to define a JavaScript object called routes which is basically just an association between URL patterns and function calls. When the request comes in for a certain URL, Express routes that to a function used to handle the call. These associations and logic are handled in web app logic and web app urls section of the code.

The final parts of the code do two things: 1) they establish the MongoDB connection so you can call it throughout the rest of the code and 2) start up the Express App server. We don’t need to modify these sections at all for our example.

You can also just skip ahead and see the finished code on Github and follow the README.md instructions to clone it into your OpenShift account. But if you do that you spoil all the fun, try instead to do the code yourself.

Importing the data

The first post in this series covered how to import the data from a JSON file of the national parks into your mondodb database and prepare it for spatial queries. Please follow those instructions to import the data into the pythonws DB into a collection called parkpoints.

Quick digression to explain Express

Express works very similar to most URL mapping schemes. You specify URLs that you want the Express framework to handle and then you also specifcy which function gets called with that URL.

So for example in our example code in server.js, the base url “/” gets mapped:

self.app.get('/', self.routes['root']);

to the function that is identified as ‘root’ in the routes object:

self.routes['root'] = function(req, res){};

So that the function associated with that attribute will be called, which in this case is a call to the MongoDB data store with a query.

We are going to define URL mappings for some basic REST like functionality to interact with our spatial MongoDB data store. The one feature we will add to this application over the Python application is the ability to POST a new location.

Modify the source code

Basic changes

First we need to load a more recent version of the MongoDB Node.JS driver and of Express. Please open package.json and add the following to your dependencies

"dependencies": {
  "mongodb" : "> 1.1.10" ,
  "express" : ">3.0.0rc4"
},

We load the most recent version of the MongoDB because it will allow for the authentication we need to enable.

We also need to 1) change the construction of Express (the old way was deprecated) and 2) add some Express middleware to the project. Middleware acts as an interceptor on requests and responses to give consitent functionality. For those from the JEE world they are like Listeners. Express basically uses the Middleware from Connect – so you can look at that documentation to understand the middleware (thanks to the #Node.js channel on Freenode for help). Since we will be posting data we are going to load the bodyParser middleware.

Please replace the line that used to say

var app = express.createServer();

to

self.app  = express();
 
//This uses the Connect frameworks body parser to parse the body of the post request  
self.app.configure(function () {
  self.app.use(express.bodyParser());
  self.app.use(express.methodOverride());
  self.app.use(express.errorHandler({ dumpExceptions: true, showStack: true }));
});

Time to code

The first function we are going to write will be to just simply return all the records in the database. In a more full featured app you would probably want to add pagination and other features to this query but we won’t be doing that today.

Let’s just remap the root URL to be a simple text explanation of the site.

self.route['root'] = function(req, res){
  res.send('You have come to the park apps web service. All the web services are at /ws/parks*. For example /ws/parks will return all the parks in the system in a JSON payload. Thanks for stopping by and have a nice day');
};

Now let’s get to the fun stuff – database queries and JSON payloads!

The first thing we have to do is change the database we are connecting to. By default we create a db with the same name as the application but in this case the mongoimport made a db called parks. Take this line at the top of server.js:

self.db = new mongodb.Db(process.env.OPENSHIFT_APP_NAME, self.dbServer, {auto_reconnect: true}); 

and change it to:

self.db = new mongodb.Db('parks', self.dbServer, {auto_reconnect: true});

We also need to modify the authentication section for MongoDB. OpenShift has authentication turned on for the database, therefore any connection has to authenticate. By default a connection from the MongoDB driver will try to authenticate against users in the DB you specify. Because we are using a different DB than the default OpenShift provided DB (which was named nodewsos), the normal admin user created at cartridge creation time is not present. Therefore on connection, we need to tell the driver to authenticate this user against the admin database. Please change the authentication line later in server.js to this:

self.db.authenticate(self.dbUser, self.dbPass, {authdb: "admin"}, function(err, res){...

Now our code is ready to talk to the DB. Let’s fix up our route to get all records. First we add a new route to the web app urls section

self.app.get('/ws/parks', self.routes['returnAllParks']);
self.routes['returnAllParks'] = function(req, res){
  self.db.collection('parkpoints').find().toArray(function(err, names) {
    res.header("Content-Type:","text/json");
    res.end(JSON.stringify(names));
  });
};

I chose to put the web services under the url /ws/parks so that we could use other parts of the URL namespace for other functionality. You can now go to your application URL (http://pythonws-.rhcloud.com/ws/parks) and you should be able to see all the documents in the DB.

Grab just one park

Next we will implement the code to get back a park given a parks uniqueID. For ID we will just use the ID generated by MongoDB on document insertion (_id). The ID looks like a long random sequence and that is what we will pass into the URL.

return a specific park given it’s mongo _id

self.app.get('/ws/parks/park/:id', self.routes['returnAPark']);
 
 //find a single park by passing in the objectID to the URL
  self.routes['returnAPark'] = function(req, res){
      var BSON = mongodb.BSONPure;
      var parkObjectID = new BSON.ObjectID(req.params.id);
      self.db.collection('parkpoints').find({'_id':parkObjectID}).toArray(function(err, names){
              res.header("Content-Type:","application/json");
              res.end(JSON.stringify(names));
      });
  };

Here you have to use another class from the bson library – ObjectID. The actual ObjectID in MongoDB is an object and so we have to take the ID passed in on the url and create an Object from it. The ObjectID class allows us to create one of these objects to pass into the query.

Other than that the code is the same as above.

This little snippet also shows an example of grabbing part of the URL and passing it to a function. I explained this concept above but here we can see it in practice.

Time for the spatial query

Here we do a query to find national parks near a lattitude longitude pair

self.app.get('/ws/parks/near', self.routes['returnParkNear']);
 
//find parks near a certain lat and lon passed in as query parameters (near?lat=45.5&lon=-82)
self.routes['returnParkNear'] = function(req, res){
 
  var lat = parseFloat(req.query.lat);
  var lon = parseFloat(req.query.lon);
 
  self.db.collection('parkpoints').find( {"pos" : {$near:[lon,lat]}}).toArray(function(err,names){
    res.header("Content-Type:","application/json");
    res.end(JSON.stringify(names));
  });
};

This piece of code shows how to get request parameters from the URL. We capture the lat and lon from the request url and then cast them to floats to use in our query. Remember, everything in a URL comes across as a string so it needs to be converted before being used in the query. In a production app you would need to make sure that you were actually passed strings that could be parsed as floating point numbers. But since this app is just for demo purposes I am not going to show that here.

Once we have the coordinates, we pass them in the the query just like we did from the command line MongoDB client. The results come back in distance order from the point passed into the query. Remember, the ordering of the coordinates passed into the query need to match the ordering of the coordinates in your MongoDB collection.

Regex query with spatial goodness

The final piece of query code we are going to write allows for a query based both on the name and the location of interest.

self.app.get('/ws/parks/name/near/:name', self.routes['returnParkNameNear']);
 
self.routes['returnParkNameNear'] = function(req, res){
  var lat = parseFloat(req.query.lat);
  var lon = parseFloat(req.query.lon);
  var name = req.params.name;
 
   self.db.collection('parkpoints').find( {"Name" : {$regex : name, $options : 'i'}, "pos" : { $near :[lon,lat]}}).toArray(function(err,names){     
    res.header("Content-Type:","application/json");
    res.end(JSON.stringify(names));
  });      
};

Just like the example above we parse out the lat and lon from the URL query parameters. In looking at my architecture I do think it might have been better to add the name as a query parameter as well, but this will still work for this article. We grab the name from the end of the URL path and then use it in our regular expression. I added “$options : ‘i'” to make the regex case-insenstive. I then use the regex to search against the Name field in the document collection and do a geo search against the pos field. Again, the results will come back in distance order from the point passed into the query.

This time we do an insert

For the example today we are just going to insert new parks. To test this you will need to use some HTTP client or tester that allows you to POST and add a message body. There are several for Firefox extensions for testing – I used HTTPRequester. You want to make sure to set the mime type to application JSON and the payload should look someting like this:

{ "name" : "My Park", "lat" : 52.5, "lon" : 52.5} 

Which basicallys says the park name is “My Park” and the lat and lon give the coordinates. The code to handle the payload looks like this:

The url mapper:

self.app.post('/ws/parks/park', self.routes['postAPark']);

Notice that we are now responding to a POST event (app.post) and so if you hit this url with a GET it will return a 404. That comes automagically from the Express framework.

Now the business logic:

self.routes['postAPark'] = function(req, res){
  var name = req.body.name;
  var lat = req.body.lat;
  var lon = req.body.lon;
 
  self.db.collection('parkpoints').insert({'Name' : name, 'pos' : [lon,lat ]}), function(result){
    //we should have caught errors here for a real app
    res.end('success');
  });
};

Once again we should have checked the payload to insure it had “valid” content and we should have also checked to make sure there was no error submitting the data. Again, since this is a small sample app I am not going to show all that today. But as you can see it is quite easy to insert a new park in the parkpoints collection. It will be indexed on insertion and be ready for our spatial queries above.

Conclusion

And with that we have wrapped up our little web service code – simple and easy using Node, Express, and MongoDB. Again, there are some further changes required for going to production, such as request parameter checking, maybe better URL patterns, exception catching, and perhaps a checkin URL – but overall this should put you well on your way. There are examples of:

  • Using Express to write some nice REST style services in Node.JS
  • Various methods to get URL information so you can use it in your code
  • How to interact with your MongoDB in Node.JS using Mongo and BSON libraries
  • Getting spatial data out of your application

Give it all a try on OpenShift and drop me a line to show me what you built. I can’t wait to see all the interesting spatial apps built by shifters.

Categories
MongoDB, Node.js, OpenShift Online
Tags
Comments are closed.