node.js, socket.io, and redis: Beginners Tutorial – Server side

Download the src(github)
View the demo

This project has been updated since this was published. The project now includes Express 4.2, Socket.IO 1.2, newer version of Redis, promises, and testing. It is all around better and more up to date. You can read the new post here.

This is a simple application that I have used to try my hand at node.js, socket.io, and redis. The idea came from a co-worker who asked me if I could build an application to collect the votes of our other co-workers to determine where to eat. An initial inspiration came from geoloqi through a game they created. It introduced me to using node.js, socket.io, and redis together to keep multiple browsers in sync.

I had a period of rethinking the entire app design as node.js and redis are both new and different concepts. Node.js requires you to think about how and when(<- this is important) each statement will run. Fenn Bailey has a great post about how ifs and loops work completely different in node (or not at all depending on your point of view). You can get a lot of nulls out of loops if you do not plan out correctly. Redis is a simple and fast key-value store. It has some useful built-in types, but what it doesn’t have is a schema. You cannot map out a row with an id and multiple fields. You have to map your data to keys and sets. I will cover all of my node and redis issues and solutions in this post.

node.js

I am not really doing a lot with node.js here. Node is filling in the server role to give me a platform that I can build from. Although technically that’s what every node.js application does. My node.js parts just listen on 8080 and returns “Hello Socket” on any request. I could have pulled in the HTML to serve from node, but I am just letting Apache serve it. All the good parts are in socket.io which connects through javascript.

One thing you will run into is that node doesn’t have a daemon. When you are debugging this is not an issue as you just run it from the command line and let go. You can run the process in the background, but that is prone to issues. I have found forever as a great way to keep a node.js app running.

The only ‘true node.js’ code in my app.js:

var app = require('http').createServer(handler)
  , io = require('socket.io').listen(app, {origins: '*:*'});

var redis = require("redis");
client = redis.createClient();

app.listen(8080);

function handler (req, res) {
	//I don't know if I need this
    res.writeHead(200);
    res.end("Hello Socket");
  
}

Redis

I am first going to cover redis and my data design and then move on to socket.io. Redis is a self described “open source, advanced key-value store”. This is a very different concept from relational databases. The easiest way to think about is that in a RDBS you create rows which can have mulitple fields, compared to redis where each row is a field. For example in a RDBS if you want to hold the URL for an image you would add a text/varchar field in your row. In redis you would create a user:userid:img key which would hold the data. The use of no schema key value store was one reason I wanted to use redis, as I have never used something like this to store application data. Another reason was speed. Redis is an in memory store. Simple value calls are fast. This application does not have a lot of data and that is a good use case for redis. If I had bigger datasets I would probably use something else.

Data Structure

The data structure really just breaks down to users and votes. Each user is part of an area (the area is set on the client side by adding a hash to the URL) and gets a vote in each area. That’s it. Here is what a user would look like if you collected all the keys.

  • id – [area]:users:[username]
  • image URL – [area]:users:[username]:img
  • vote – [area]:users:[username]:vote

The id is stored in a set with a key of [area]:users. If given an area you would be able to list all the users including an image, and that users vote. The id is also stored in a another set that tracks the votes with a key of [area]:votes. Again if given an area you can list all the votes along with the user that gave the vote. That is it, the entire data structure.

I also use the expiration method of redis to expire all the keys. Each value is set to expire two hours later. This allows the application to reuse areas for different meals (lunch and dinner). It also stops the store from growing too large. This works great for all the user values. It does not work so great for sets. When I try to expire a set, redis turns the set into a value key. I would only have one of the values in the set and it would overwrite the current value when I tried to set anything to it. This is not what I wanted.

I will note that I am using redis 1.2 which is the version in the default repos of my version of Ubuntu. I know that I can download and compile a new version or use a ppa, but I did find a way to make it work how I wanted.

To ensure that the sets did not live on forever is that I created another key called timer for each area ([area]:votes:timer) and set an expiration on that. That key was then put into a set called expireKeys that is looped through every 10 minutes to see if the timers have expired and then delete the matching sets. It is not a lot of overhead as it only runs every ten minutes and it makes sure that the redis store stays clean.

Let’s look at some code on how to save a user to redis in node.js. The client variable is created and set earlier in the script as var client = redis.createClient();

function setUser(username, img, area, expire){
	client.set(area+':users:' + username, username);
	client.expire(area+':users:' + username, expire);
	client.set(area+':users:' + username + ':img', img);
	client.expire(area+':users:' + username + ':img', expire);
	//set a timer
	client.set(area+':users:timer', expire);
	client.expire(area+':users:timer', expire);
	client.sadd(area+':users', area+':users:' + username);
	//add the set to the expire set
	client.sadd('expireKeys', area+':users');
};

This is pulled out into a function and is called when a user addition event is fired. It is very straightforward, key is created with set and then expired with expire. The identity (area:users:username) is then added to the set area:users with sadd.

Setting a vote is very similar.

function setVote(username, area, fs, expire){
	client.set(area+':users:' + username + ':vote', JSON.stringify(fs));
	client.sadd(area+':votes', area+':users:' + username);
	//add the set to the expire set
	client.sadd('expireKeys', area+':votes');
	//set a timer
	client.set(area+':votes:timer', expire);
	client.expire(area+':votes:timer', expire);
};

There is not history with this implementation. If you voted before your vote is overwritten in the set as each voter can only be in the set once. The same is true with users. This does open up impersonation and vote tampering, but with this application I view that as more of a people problem than a technical problem. I also do not think it is a serious issue as all votes and users are gone in a couple of hours.

I will cover the function that checks for expired keys now. This demonstrates how things flow in node.js.

function checkExpires(){
	//grab the expire set
	client.smembers('expireKeys', function(err, keys){
		if(keys != null){
			keys.forEach(function(key){
				console.log(key);
				client.get(key+':timer', function(err, timer){
					//grab the timer
					if(timer != null){
						//timer exists check the ttl on it
						client.ttl(key+':timer', function(err, ttl){
							//the ttl is two hours and if it is under an hour
							//and a half we delete it
							if(ttl < 5400){
								console.log(ttl);
								client.del(key);
								client.srem('expireKeys', key);
							}
						});
					}else{
						//the timer is gone delete the key
						client.del(key);
						client.srem('expireKeys', key);
					}
				})
			});
		}
	});
};

There are a lot of brackets. This is because in node everything is done in a callback. You make a request or in this case a redis call and the value is returned in a callback. You do not write var ttl = client.ttl(key+’:timer’);, you write client.ttl(key+’:timer’, function(err, ttl){//use ttl here});. In the function above all the tabs and brackets are there because it is running a callback inside of a callback inside of a callback. When we first we get the keys, this will be returned as an array. If we try to set a variable to this in the current scope it will always return null. This is different from writing javascript on the client side. The data you want only lives in the callback, so everything happens in the callback. Callbacks are not anything new to javascript and if you can adjust to this thinking, writing node apps becomes a whole lot easier.

Socket.io

Socket.io makes it trivial to connect multiple browsers and have them react in realtime with each other. There are many chat demos out there that show how to take a message from one browser and send it to all the others immediately. That is all this application is doing. When someone votes it is broadcast out to everyone else in the same area. In the context of node.js, socket.io is where all the events are created and handled. Socket.io has all the wiring to respond to the clients requests.

First thing to do is listen for connections. Because this is node everything else happens in the callback from this. This application is listen on the /users path.

var users = io.of('/users').on('connection', function (socket) { //everything here });

We get a socket variable out of this that should be our connection to the client. We now listen for specific events from the client. Socket.io makes this really easy. Emits send a message and ‘on’ catches the message. In this application we have three events we created and one built-in event; add, addVote, getVotes, and disconnect. Socket.io also has support for rooms. This allows you to group your connections into specific, um… well, groups. You can then send a broadcast to just connections in that room. Just like everything else socket.io does this is very easy.

Here is the add event.

	socket.on('add', function(username, img, area){
	if(!usernameExists(area, username)){
		setUser(username, img, area, 7200);
	}
		//var test = new User(username, img, area, socket.id);
		socket.join(area);
		user = new User(username, img, area, socket.id);
		socket.set('user', user);

		});

The socket.on method is where you specify the event name and what parameters it is expecting. In this method we store the username and img URL in redis with the setUser function and then socket.join the area which was specified.

Here is addVote.

	socket.on('addVote', function(fs){
		if(user !== null){
				setVote(user.username, user.area, fs, 7200);
				io.of('/users').in(user.area).emit('vote', {username: user.username, img: user.img, fs: fs});
				//socket.emit('vote', {username: user.username, img: user.img, fs: fs});
		}
	});

The fs parameter is a FourSquare venue object. We store the vote in redis with the current users information (which was set in the add event). We then broadcast it to everyone in the current area. With socket.io it is one line of code. This will emit an event to everyone that has added a user in this area.

Here is getVotes.

	socket.on('getVotes', function(){
		var area = user.area;
			client.smembers(area+':votes', function(err, votes){
				if(votes != null){
					votes.forEach(function(key){
						client.get(key, function(err, username){
							client.get(key + ':vote', function(err, vote){
								client.get(key + ':img', function(err, img){
									socket.emit('vote', {username: username, img: img, fs: JSON.parse(vote)});
								});
							});
						});
					});
				}
			});
	});

Notice all the brackets. We are calling callbacks inside of callbacks. It will grab one bit of info (for example the img of the user that made the vote) and send it to a callback. When we get 5 callbacks in we finally emit the info back to the asking client. My first iteration through when writing this I was trying to populate an array with all this info and make one emit call. I kept getting nulls everytime. I finally quit trying to fight node.js and work with how it was architected by using callbacks.

Summary

I am sure there are better ways to do this, but it does work. This application has helped me think differently about how to design an app. Key value stores and evented I/O is something I have never worked with before. Hopefully I can jumpstart someone down these paths by solving a problem. I will cover the client side in the next post.

Facebook SDK Login for Zend Framework Tutorial

Download the src(github)
View the demo

I previously had written about using the Facebook SDK inside of Zend Framework. It did not go very deep into actually using the Facebook SDK. In this post I will show you how to use it as a login system. But wait there is more. I will also show you how to use Twitter, Google, and essentially any other service you want. I use the abstraction that Zend Framework gives you for authentication and add one more layer to that so that you can easily plug in new services to authenticate to.

Zend Framework Authentication
A successful login.

Zend Framework Setup

This is a basic Zend Framework 1.11 install. I did make a few modifications to it. Because one of the authentication methods is OpenId using Google you have to patch the Zend_Openid_Consumer. You can see this why in the Zend Framework issue browser. The other is a way to read attributes from OpenID. This was written by Chris Bisnett. That is the only modifications to the core of Zend Framework. You will want to make sure that you have the most current version of the Facebook SDK. The included Facebook SDK is 3.1.1.

High Level Overview

I will first walk you through how it authenticates any provider and then specifically discuss each that I have. Basically this is what the application does: Continue reading “Facebook SDK Login for Zend Framework Tutorial”

Google Event Tracking

I will talk about event tracking in Google Analytics today. Almost everyone has heard of Google Analytics and I am guessing you are currently using it to track page views. All you have to do is pop a little javascript on your page and you have hundreds of ways to view the traffic your site receives. If you are not using event tracking though, you are missing out on quite a few data points. For a quick primer look at Google’s documentation.

Events

I won’t go into the general idea of tracking events. You have to decide what constitutes an ‘event’ that you want to track. A quick search turns of thousands of blogs that will give you ideas. I want to specifically talk about specific use cases. The first is a one page javascript application. When you are only tracking page views you will miss 90% of the interaction on the page. You have to piggyback on events to see what is happening in the app. The other case is errors. Javascript is client side so if it breaks you do not see it. Event tracking can track these for you.

App Interactions

I will use RunBrowser as my example app. It is a single HTML page javascript app. I used the event listeners on buttons to log events to Google. I did this inside of my AppController object. You can specify the category, action, label, and value for each event. I mainly just used category and action as it was a small app. I tied the main Start button to a Runbrowse category and Start action. Then I tied the save or clear button presses to the Runbrowse category and Save or Clear actions respectively. At this point I can view how many people started runs and then eventually saved them. This data would be lost without event tracking.

Here is a report called Events Flow:
Events Flow

You see that 35 people from the US viewed the page during the selected dates. 4 of them viewed a saved run and 31 started a new run. 20 of those 31 paused it to hit clear or save, I did not extend the image past that, but you would be able to see how many people eventually saved their run (not the actual run and the data associated just the event of saving it). Out the 11 that did not pause 3 had errors. Without event tracking I would have only seen 35 visits from the US and missed out on all this other data.

Errors!

Tracking errors with events is a great way to see what is happening out in the wild. As we saw in the events flow report I had some errors come up. In my error handler for GPS I track the category as Error and the action as the error code. The error codes match what is returned from the location object according to the API. The three values map to 1= PERMISSION_DENIED, 2 = POSITION_UNAVAILABLE, and 3 = TIMEOUT. I can view my Error category and get a rundown of all the errors on my page.
Errors

This report shows I had 12 events where permission was not given to track the location and 4 timeouts. The other event is the clear modal after the error. You could have client side issues and errors and most people will not send a bug report back to you. With event tracking of errors you can see exactly how many errors and hopefully why. You can also tie in other information like browser and OS version to see if you have a specific bug that only affects a subset of users.

Unobtrusive Event Tracking

I found a great jQuery plugin to do Google Event tracking. Because it is jQuery you don’t have to worry about cross browser issues. It allows you to easily add event tracking to an existing page or to on a new page. It attaches right to the click event handler so you don’t have to muddy up your HTML with a lot of onclick attributes. It has the ability to use HTML5 data-* attributes to set category, action, label, and value. It has many options that should be able to suit your needs. It is also hosted on GitHub.

HTML 5 run tracking application

Download the src(github)
View the demo

The github repo has been updated. Some of the code will not match what is currently in the repo.

An idea that I have had for awhile is a RunKeeper like application that runs completely in the browser. This is not due to any inherit issues with the RunKeeper application, just more of a proof-of-concept. HTML 5 makes this easy. We can save old runs and run without an Internet connection making it almost a native app.

The only phone I have tested this with is an iPhone 4S as that is the only phone I have access to. It should work with an iPhone 4 and possibly a 3GS, but I cannot say for sure. Android should work as well as it should support everything I am doing with HTML 5 and javascript. Desktops can also try it out if you are using any modern browser: Chrome, FireFox, or Safari. You won’t get a path and possibly get an error if your browser cannot get a location. I have tested it in IE and it does not work. I have no plans to try and make it work as the only place I was trying to make it work was the iPhone. Although not supporting IE has made the code a lot cleaner.

RunKeeper in the browser: RunBrowser

HTML 5 – almost a native app

HTML 5 gives us many new methods to make a webpage an application. Before HTML 5 there were many workarounds to make a webpage do what we wanted it to. Now they are built right into the browser. The first thing I am going to cover is application cache. Continue reading “HTML 5 run tracking application”

Facebook SDK and Backbone – Final Post

Download the src(github)
View the demo

This really is the final installment of this. I have shown how to use the router and views to make a site that is only one page of HTML, but still has back button functionality. Each view is tied to a Facebook API query in which I load different elements on to the page. I have a few different ways in the code that you can do this. I would like to say it is for instructional purposes, but it is actually because I was trying different methods and I did not want to go back through and update all the other views and routes.

I will show how to deal with paged data and some other tricks

Paged Facebook Data

Let’s use photos for example. To see what would be returned for an API call to /me/photos we will use the Graph API explorer. You will see that there are two top level keys data and paging. Inside of data is all the information about each photo. In the lofPictureView I loop through the photos adding them to the page

         for (photo in this.model.data)
    		{
    			var test = "<li>";
    			test += "<img src=\"" + this.model.data[photo].source + "\" class=\"fb-pics\">";
    			if(this.model.data[photo].comments){
    				test += "COMMENTS: " + this.model.data[photo].comments.data.length;
    			}
    			test += "</li>";

    			this.jQel.children('ul').append(test);
    		}

I create a list item that has an img tag and then I check for comments and let you know how many there are. If you wanted to add different information you can look through the object in the Graph API explorer and see what is available to use. Now let’s look at the paging data. This is pretty straight forward. The previous attribute has the URL for previous photos and the next has the URL for the next photos. First thing I do is see if these attributes are present and then I create a button for them.

 		
    		if(this.model.paging.previous)
    		{
	    		var prevButton = document.createElement('button');
	    		prevButton.innerHTML = 'PREVIOUS';
	    		prevButton.id = 'prevPage';
	    		this.jQel.children('div').append(prevButton);
    		}
    		
    		if(this.model.paging.next)
    		{
	    		var nextButton = document.createElement('button');
	    		nextButton.innerHTML = 'NEXT';
	    		nextButton.id = 'nextPage';
	    		this.jQel.children('div').append(nextButton);
    		}

Earlier in the Backbone view object we created click events for a button that has the id of prevPage and a button that has the id of nextPage that call functions with the same name as each id.

		events: {
			'click button#nextPage' : 'nextPage',
			'click button#prevPage'	: 'prevPage'
		},

Inside of these functions is the first gotcha. I am using jQuery’s getJson function. We want this to be JSONP and not plain old JSON. Why JSONP? Remy Sharp explains it best, “JSONP is script tag injection, passing the response from the server in to a user specified function”. We want to use this data in an anonymous function call and we need it to be JSONP. Using jQuery’s documentation we see that if we add callback=? the request will be treated as JSONP. In my functions we add callback=? and we get an object back that we can use.

    	nextPage: function(){
    		var This = this;
			$.getJSON(this.model.paging.next + '&callback=?', function(response){
				loadPhoto(response, This.options.currUser);
			});
    	},
    	
    	prevPage: function(){
    		var This = this;
			$.getJSON(this.model.paging.previous + '&callback=?', function(response){
				loadPhoto(response, This.options.currUser);
			});
    	}

You can see that we use the data returned to load another lofPhotoView of new pictures. We also pass the cached user object through the closure This.options.currUser.

Backbone Event Target

In the lofPostView I have a list of a person’s last posts. I put an attribute of data-post-id equal to the post id so that I can retrieve it. The issue comes in when you want to have access to that element and retrieve the data-post-id attribute. Backbone passes the event as a parameter so that you can pull into your function and use it. This Stackoverflow question shows how to use it. Once you have the target you can ask for the data-post-id attribute. There is one issue with this particular setup, sometimes the click will be on a div inside the li. The div does not have the attribute so we cannot grab the data from there as it does not have the attribute. We can either put this attribute on all child elements or we can test to see if the attribute is present and if not grab it from the parent. Here is the code in the lofPostView where I do that.

    	post: function(ev){
    		if(!$(ev.target).attr('data-post-id')){
    			var postid = $(ev.target).parent('li').attr('data-post-id')
    		}else{
    			var postid = $(ev.target).attr('data-post-id');
    		}
    		window.location.hash = '/post/' + postid;
    	},

I am sure there are better ways to do what I did. If you know of one of the ways let me know. You can leave a comment or even fork the project on github.

Facebook SDK with Backbone – pt 3

Download the src(github)
View the demo

I am back to finally finish this off. We are going to dive into Backbone and how we use it to create this application. I want to note up front that this is not a great tutorial for Backbone. The reason being is that I don’t use any of the model aspects of Backbone. I use it for event binding and creating views. In this context it makes sense as our model is Facebook. Another thing to note is that this uses Backbone version 0.5.3 when as of right now it is at version 0.9. It looks like it should work after being upgraded, but I haven’t tested it yet. Well let’s get started.

Finally we are getting to the backbone of this app

Backbone

Backbone is a javascript library that creates an MVC(Model, View, Controller) structure. Earlier I said this was not a great tutorial and that is because we are only using the view and controller portions of backbone. The model is coming from Facebook. We probably could abstract this out and create a backbone model for Facebook, but I chose just to use the Facebook API directly. Continue reading “Facebook SDK with Backbone – pt 3”

Facebook SDK using only javascript – pt 2

Download the src(github)
View the demo

I know it has been awhile. Christmas came and proceeded to fill up a lot of my time. I am back, though, to finish this tutorial. We left off at the HTML skeleton. Remember this is just an outline of where we are going to put our real content. This content we will fetch from Facebook through Facebook’s SDK and then throw it up on the screen. Let’s get on to using the Facebook JavaScript SDK.

This part will cover setting up the Facebook SDK, initializing the SDK, and getting users authenticated.

Facebook SDK setup

We covered this in the last post, but I will touch on it again. We are going to use XFBML. Facebook used to have FBML, but they have deprecated this. We use XFBML to create the login button. If you look at Facebook’s Social Plugins you will see that you can implement many of them through the javascript SDK and XFBML. The last post had the old way of doing it, but here is the new updated way (it’s not much different):

<html xmlns:fb="http://ogp.me/ns/fb#">

Continue reading “Facebook SDK using only javascript – pt 2”

Facebook SDK using only javascript – pt 1

Download the src(github)
View the demo

This is a continuation of my last post. My re-immersion into NES games started me down the path of an 8-bit web browser. This project is not a full 8-bit browser, but rather just viewing Facebook. It uses javascript and only javascript. There is a skeleton of HTML elements for javscript to build the content. It uses the Facebook js SDK,  backbone.js is used to build the content, pixelize to pixelize the photos, and jquery for ease.

It uses javascript and only javascript

Setup your Facebook App

First thing you will need to do is to create an app through Facebook. Start at the Facebook Developers site. There will be a button to create a new app. A modal window will popup. Fill this out.You may have to play with the App namespace to find something that isn’t taken. Continue reading “Facebook SDK using only javascript – pt 1”

Pixelize photos with HTML 5 canvas and Javascript

Download the src(github)
View the demo

This library came about because of a my purchase of an NES. It lead me to wonder what would a browser look like on the NES? The first thing would be that all the photos would be pixelized. I wanted to use the canvas element of HTML 5. I didn’t want to go through every pixel of the photo to do the pixelization, I just wanted to shrink the image down and then blow it back up to create ‘natural’ pixelization.

What would a browser look like on the NES?

The canvas element

Canvas is a new element that was part of HTML5. Canvas opens a lot of doors in HTML5 and there are a lot of amazing examples of what is possible with the canvas element, but we are only using a small portion of it’s capabilities. We can add a canvas to a web page very easily

<canvas id="myCanvas" width="500" height="500">Fallback content</canvas>

We can now manipulate it with Javascript. First thing is to get a reference Continue reading “Pixelize photos with HTML 5 canvas and Javascript”