Node.js,, and Redis: Intermediate Tutorial – Client side (Screencast)

Download the src(github)
View the demo

Blog Post Series


Continue reading “Node.js,, and Redis: Intermediate Tutorial – Client side (Screencast)”

Node.js,, and Redis: Intermediate Tutorial – Client side

Download the src(github)
View the demo

Blog Post Series

We will begin by managing our dependencies. Let’s get started.


Bower is a package manager like npm. In much the same way as npm, Bower needs Node.js and runs on top of it. While npm focuses on packages for Node.js, Bower usually targets front-end libraries like jQuery and React. Managing front-end libraries is much better than the old way of just downloading a version and sticking it in our project. We now have full version control.

Bower uses a configuration file named bower.json. Here is that file for our project. Continue reading “Node.js,, and Redis: Intermediate Tutorial – Client side”

Node.js,, and Redis: Intermediate Tutorial – Server side (Screencast)

Download the src(github)
View the demo

Blog Post Series


Continue reading “Node.js,, and Redis: Intermediate Tutorial – Server side (Screencast)”

Node.js,, and Redis: Intermediate Tutorial – Server side

Download the src(github)
View the demo

Blog Post Series

This post has been a long time coming, about a year in fact. Although I have not been lazy. As you can probably tell from my other posts and the little blurb at the end of each post, is that I have written a book. It focuses on Node.js,Express, Passport, Socket.IO, Redis, RabbitMQ, React, Backbone, Grunt, and Ansible. All in one book! Which brings us to this post in a round about way.

One of my most popular posts is about using Node.js, Socket.IO, and Redis to build a small voting application. It is also my most forked repo on Github. After finishing the book I decided to revisit this codebase and see how it has held up over time. Unfortunately it needed some improvement. There is a law of programming that states any code that is more than three months old will be viewed as crap. Some of the improvements are upgrades to libraries (Express 4.2, Socket.IO 1.2) and React for client side rendering (I am currently working on the client side and should be ready soon). Other improvements are just better code. An example of this is using promises for data retrieval and much better testing.

You will not need to read the previous post as I will take you through the all of the code and explain what it all does. This is my longest post to date, but it does cover everything you will need to know about the server side of this project. Here is what we will cover:

  • Configuration and environment variables
  • Express 4.2+
  • Socket.IO 1.2+
  • Redis and building a Redis data structure
  • Repository pattern using dependency injection
  • Testing the repository
  • Testing Socket.IO

Let’s start at the beginning with the package.json.


The package.json file for a Node.js project is the best place to start. This defines the main file, dependencies, any scripts that can be run, and many other things. Here is the package.json for this project.  Continue reading “Node.js,, and Redis: Intermediate Tutorial – Server side”

I have a Node.js and Redis book coming out soon!

I have not been updating this blog like I should have been for about the last year. That is unfortunate. Although I feel I have a good reason. I have been writing a book! You can currently see the pre-order page at Packt Publishing.

This book is almost 300 pages that will start the reader from an empty folder to a working, scalable application that is published to Amazon Web Services. Some of the topics covered are:

  • Express 4.0+
  • Authentication with Passport
  • Socket.IO 1.0+ using the same authentication
  • Message Queues using RabbitMQ
  • Shared application state using Redis
  • Building a real-time frontend using React and Backbone
  • Grunt as a JavaScript build tool
  • Testing with nodeunit and coverage with Istanbul
  • Deployment to Amazon Web Services using Ansible
  • Troubleshooting and debugging Node.js using node-inspector and webkit-devtools-agent

If you have enjoyed any of my posts so far (and even if you did not) you will like this book. You will not need expert knowledge in any of these frameworks to benefit from this book. It will show you what you need to know about each framework and most importantly, how all the frameworks work together in a single application. Lots of examples with code are included with the book.
Myself, being biased, definitely recommends this book. There is something in this book for everyone no matter what there experience level is.

WebRTC – Using your Webcam for video in HTML Tutorial

Download the src(github)
View the demo

WebRTC is project to bring Real Time Communications to the browser. It is designed to allow video/voice chat with the browser without plugins. Because you technically have to have access to the webcam to do video chat, WebRTC gives you a javascript API to the webcam. This is what I am going to show you how to use. I am not going to go into the audio portion or how to create a real time communication link. WebRTC is still very much in beta and most browsers use a vendor prefix to get access to the webcam. I will show you what I did to create my webcam javascript abstraction library.

Create a Webcam Library

I created a small shim/abstraction library to make getting and using the webcam easier. I used what I learned from creating CreateMyPhotobooth (this is a small plug, but I am relaying what I learned from this). I have the project on github with a demo, grunt build info, and tests.

Use of the Library

When you include the library you will have access to the wcvj (WebCamVidJa) object. The first function is videoIsSupported. Continue reading “WebRTC – Using your Webcam for video in HTML Tutorial”

Facebook and FourSquare API checkin calls with SVG in HTML tutorial

Download the src(github)
View the demo

This has been an idea in my head for awhile now. The app will get your checkins from Facebook and FourSquare and will show you on a map which states you have been to. It is very similar to the other apps I have created. Although I do use a new skill, manipulating SVGs.

I want to give a quick thanks to Theshibboleth who created the SVG of the US. I downloaded it from Wikipedia.


SVG stands for Scalable Vector Graphics. The simplest definition is that it is an XML file that describes how to build an image. If you would like to know more as always you can use Wikipedia or do a quick Google search. SVG has support in HTML5 for inline and loading. I am using inline SVG for this project. To do that all you have to do is copy out all the XML from the SVG file and paste it into your page. This lets you treat the SVG just like HTML. By that I mean that you can apply CSS classes to change the look and use javascript to attach event listeners and/or modify attributes on the fly. Continue reading “Facebook and FourSquare API checkin calls with SVG in HTML tutorial”

node.js,, and redis: Beginners Tutorial – Client side

Download the src(github)
View the demo

This demo has been updated since this was written. The new version uses Express 4.2.0 and Socket.IO 1.1.0. Some of the statements may not match what is currently in github.

In the previous post I discussed the server side. This post will cover client side. The server side has a lot less code because it only worries about taking the requests and putting them into redis and then sending them back out. The client has the all the display logic. Let’s get started.

The great thing about is that the client side is just as easy as the server side. The same three basic actions are there, connect, on, and emit. The basic idea is that one side emits to the on event of the other side. will keep track of the connection for you. On the the side that listens you get whatever you passed through the connection. Continue reading “node.js,, and redis: Beginners Tutorial – Client side”

node.js,, and redis: Beginners Tutorial – Server side

Download the src(github)
View the demo

This project has been updated since this was published. The project now includes Express 4.2, Socket.IO 1.2, newer version of Redis, promises, and testing. It is all around better and more up to date. You can read the new post here.

This is a simple application that I have used to try my hand at node.js,, and redis. The idea came from a co-worker who asked me if I could build an application to collect the votes of our other co-workers to determine where to eat. An initial inspiration came from geoloqi through a game they created. It introduced me to using node.js,, and redis together to keep multiple browsers in sync.

I had a period of rethinking the entire app design as node.js and redis are both new and different concepts. Node.js requires you to think about how and when(<- this is important) each statement will run. Fenn Bailey has a great post about how ifs and loops work completely different in node (or not at all depending on your point of view). You can get a lot of nulls out of loops if you do not plan out correctly. Redis is a simple and fast key-value store. It has some useful built-in types, but what it doesn’t have is a schema. You cannot map out a row with an id and multiple fields. You have to map your data to keys and sets. I will cover all of my node and redis issues and solutions in this post.


I am not really doing a lot with node.js here. Node is filling in the server role to give me a platform that I can build from. Although technically that’s what every node.js application does. My node.js parts just listen on 8080 and returns “Hello Socket” on any request. I could have pulled in the HTML to serve from node, but I am just letting Apache serve it. All the good parts are in which connects through javascript.

One thing you will run into is that node doesn’t have a daemon. When you are debugging this is not an issue as you just run it from the command line and let go. You can run the process in the background, but that is prone to issues. I have found forever as a great way to keep a node.js app running.

The only ‘true node.js’ code in my app.js:

var app = require('http').createServer(handler)
  , io = require('').listen(app, {origins: '*:*'});

var redis = require("redis");
client = redis.createClient();


function handler (req, res) {
	//I don't know if I need this
    res.end("Hello Socket");


I am first going to cover redis and my data design and then move on to Redis is a self described “open source, advanced key-value store”. This is a very different concept from relational databases. The easiest way to think about is that in a RDBS you create rows which can have mulitple fields, compared to redis where each row is a field. For example in a RDBS if you want to hold the URL for an image you would add a text/varchar field in your row. In redis you would create a user:userid:img key which would hold the data. The use of no schema key value store was one reason I wanted to use redis, as I have never used something like this to store application data. Another reason was speed. Redis is an in memory store. Simple value calls are fast. This application does not have a lot of data and that is a good use case for redis. If I had bigger datasets I would probably use something else.

Data Structure

The data structure really just breaks down to users and votes. Each user is part of an area (the area is set on the client side by adding a hash to the URL) and gets a vote in each area. That’s it. Here is what a user would look like if you collected all the keys.

  • id – [area]:users:[username]
  • image URL – [area]:users:[username]:img
  • vote – [area]:users:[username]:vote

The id is stored in a set with a key of [area]:users. If given an area you would be able to list all the users including an image, and that users vote. The id is also stored in a another set that tracks the votes with a key of [area]:votes. Again if given an area you can list all the votes along with the user that gave the vote. That is it, the entire data structure.

I also use the expiration method of redis to expire all the keys. Each value is set to expire two hours later. This allows the application to reuse areas for different meals (lunch and dinner). It also stops the store from growing too large. This works great for all the user values. It does not work so great for sets. When I try to expire a set, redis turns the set into a value key. I would only have one of the values in the set and it would overwrite the current value when I tried to set anything to it. This is not what I wanted.

I will note that I am using redis 1.2 which is the version in the default repos of my version of Ubuntu. I know that I can download and compile a new version or use a ppa, but I did find a way to make it work how I wanted.

To ensure that the sets did not live on forever is that I created another key called timer for each area ([area]:votes:timer) and set an expiration on that. That key was then put into a set called expireKeys that is looped through every 10 minutes to see if the timers have expired and then delete the matching sets. It is not a lot of overhead as it only runs every ten minutes and it makes sure that the redis store stays clean.

Let’s look at some code on how to save a user to redis in node.js. The client variable is created and set earlier in the script as var client = redis.createClient();

function setUser(username, img, area, expire){
	client.set(area+':users:' + username, username);
	client.expire(area+':users:' + username, expire);
	client.set(area+':users:' + username + ':img', img);
	client.expire(area+':users:' + username + ':img', expire);
	//set a timer
	client.set(area+':users:timer', expire);
	client.expire(area+':users:timer', expire);
	client.sadd(area+':users', area+':users:' + username);
	//add the set to the expire set
	client.sadd('expireKeys', area+':users');

This is pulled out into a function and is called when a user addition event is fired. It is very straightforward, key is created with set and then expired with expire. The identity (area:users:username) is then added to the set area:users with sadd.

Setting a vote is very similar.

function setVote(username, area, fs, expire){
	client.set(area+':users:' + username + ':vote', JSON.stringify(fs));
	client.sadd(area+':votes', area+':users:' + username);
	//add the set to the expire set
	client.sadd('expireKeys', area+':votes');
	//set a timer
	client.set(area+':votes:timer', expire);
	client.expire(area+':votes:timer', expire);

There is not history with this implementation. If you voted before your vote is overwritten in the set as each voter can only be in the set once. The same is true with users. This does open up impersonation and vote tampering, but with this application I view that as more of a people problem than a technical problem. I also do not think it is a serious issue as all votes and users are gone in a couple of hours.

I will cover the function that checks for expired keys now. This demonstrates how things flow in node.js.

function checkExpires(){
	//grab the expire set
	client.smembers('expireKeys', function(err, keys){
		if(keys != null){
				client.get(key+':timer', function(err, timer){
					//grab the timer
					if(timer != null){
						//timer exists check the ttl on it
						client.ttl(key+':timer', function(err, ttl){
							//the ttl is two hours and if it is under an hour
							//and a half we delete it
							if(ttl < 5400){
								client.srem('expireKeys', key);
						//the timer is gone delete the key
						client.srem('expireKeys', key);

There are a lot of brackets. This is because in node everything is done in a callback. You make a request or in this case a redis call and the value is returned in a callback. You do not write var ttl = client.ttl(key+’:timer’);, you write client.ttl(key+’:timer’, function(err, ttl){//use ttl here});. In the function above all the tabs and brackets are there because it is running a callback inside of a callback inside of a callback. When we first we get the keys, this will be returned as an array. If we try to set a variable to this in the current scope it will always return null. This is different from writing javascript on the client side. The data you want only lives in the callback, so everything happens in the callback. Callbacks are not anything new to javascript and if you can adjust to this thinking, writing node apps becomes a whole lot easier. makes it trivial to connect multiple browsers and have them react in realtime with each other. There are many chat demos out there that show how to take a message from one browser and send it to all the others immediately. That is all this application is doing. When someone votes it is broadcast out to everyone else in the same area. In the context of node.js, is where all the events are created and handled. has all the wiring to respond to the clients requests.

First thing to do is listen for connections. Because this is node everything else happens in the callback from this. This application is listen on the /users path.

var users = io.of('/users').on('connection', function (socket) { //everything here });

We get a socket variable out of this that should be our connection to the client. We now listen for specific events from the client. makes this really easy. Emits send a message and ‘on’ catches the message. In this application we have three events we created and one built-in event; add, addVote, getVotes, and disconnect. also has support for rooms. This allows you to group your connections into specific, um… well, groups. You can then send a broadcast to just connections in that room. Just like everything else does this is very easy.

Here is the add event.

	socket.on('add', function(username, img, area){
	if(!usernameExists(area, username)){
		setUser(username, img, area, 7200);
		//var test = new User(username, img, area,;
		user = new User(username, img, area,;
		socket.set('user', user);


The socket.on method is where you specify the event name and what parameters it is expecting. In this method we store the username and img URL in redis with the setUser function and then socket.join the area which was specified.

Here is addVote.

	socket.on('addVote', function(fs){
		if(user !== null){
				setVote(user.username, user.area, fs, 7200);
				io.of('/users').in(user.area).emit('vote', {username: user.username, img: user.img, fs: fs});
				//socket.emit('vote', {username: user.username, img: user.img, fs: fs});

The fs parameter is a FourSquare venue object. We store the vote in redis with the current users information (which was set in the add event). We then broadcast it to everyone in the current area. With it is one line of code. This will emit an event to everyone that has added a user in this area.

Here is getVotes.

	socket.on('getVotes', function(){
		var area = user.area;
			client.smembers(area+':votes', function(err, votes){
				if(votes != null){
						client.get(key, function(err, username){
							client.get(key + ':vote', function(err, vote){
								client.get(key + ':img', function(err, img){
									socket.emit('vote', {username: username, img: img, fs: JSON.parse(vote)});

Notice all the brackets. We are calling callbacks inside of callbacks. It will grab one bit of info (for example the img of the user that made the vote) and send it to a callback. When we get 5 callbacks in we finally emit the info back to the asking client. My first iteration through when writing this I was trying to populate an array with all this info and make one emit call. I kept getting nulls everytime. I finally quit trying to fight node.js and work with how it was architected by using callbacks.


I am sure there are better ways to do this, but it does work. This application has helped me think differently about how to design an app. Key value stores and evented I/O is something I have never worked with before. Hopefully I can jumpstart someone down these paths by solving a problem. I will cover the client side in the next post.

Google Event Tracking

I will talk about event tracking in Google Analytics today. Almost everyone has heard of Google Analytics and I am guessing you are currently using it to track page views. All you have to do is pop a little javascript on your page and you have hundreds of ways to view the traffic your site receives. If you are not using event tracking though, you are missing out on quite a few data points. For a quick primer look at Google’s documentation.


I won’t go into the general idea of tracking events. You have to decide what constitutes an ‘event’ that you want to track. A quick search turns of thousands of blogs that will give you ideas. I want to specifically talk about specific use cases. The first is a one page javascript application. When you are only tracking page views you will miss 90% of the interaction on the page. You have to piggyback on events to see what is happening in the app. The other case is errors. Javascript is client side so if it breaks you do not see it. Event tracking can track these for you.

App Interactions

I will use RunBrowser as my example app. It is a single HTML page javascript app. I used the event listeners on buttons to log events to Google. I did this inside of my AppController object. You can specify the category, action, label, and value for each event. I mainly just used category and action as it was a small app. I tied the main Start button to a Runbrowse category and Start action. Then I tied the save or clear button presses to the Runbrowse category and Save or Clear actions respectively. At this point I can view how many people started runs and then eventually saved them. This data would be lost without event tracking.

Here is a report called Events Flow:
Events Flow

You see that 35 people from the US viewed the page during the selected dates. 4 of them viewed a saved run and 31 started a new run. 20 of those 31 paused it to hit clear or save, I did not extend the image past that, but you would be able to see how many people eventually saved their run (not the actual run and the data associated just the event of saving it). Out the 11 that did not pause 3 had errors. Without event tracking I would have only seen 35 visits from the US and missed out on all this other data.


Tracking errors with events is a great way to see what is happening out in the wild. As we saw in the events flow report I had some errors come up. In my error handler for GPS I track the category as Error and the action as the error code. The error codes match what is returned from the location object according to the API. The three values map to 1= PERMISSION_DENIED, 2 = POSITION_UNAVAILABLE, and 3 = TIMEOUT. I can view my Error category and get a rundown of all the errors on my page.

This report shows I had 12 events where permission was not given to track the location and 4 timeouts. The other event is the clear modal after the error. You could have client side issues and errors and most people will not send a bug report back to you. With event tracking of errors you can see exactly how many errors and hopefully why. You can also tie in other information like browser and OS version to see if you have a specific bug that only affects a subset of users.

Unobtrusive Event Tracking

I found a great jQuery plugin to do Google Event tracking. Because it is jQuery you don’t have to worry about cross browser issues. It allows you to easily add event tracking to an existing page or to on a new page. It attaches right to the click event handler so you don’t have to muddy up your HTML with a lot of onclick attributes. It has the ability to use HTML5 data-* attributes to set category, action, label, and value. It has many options that should be able to suit your needs. It is also hosted on GitHub.