node.js, socket.io, and redis: Beginners Tutorial (Screencast)

Download the src(github)
View the demo

This project has been updated since this was published. The project now includes Express 4.2, Socket.IO 1.2, newer version of Redis, promises, and testing. It is all around better and more up to date. You can read the new post here.

node.js, socket.io, and redis: Beginners Tutorial – Client side

Download the src(github)
View the demo

This demo has been updated since this was written. The new version uses Express 4.2.0 and Socket.IO 1.1.0. Some of the statements may not match what is currently in github.

In the previous post I discussed the server side. This post will cover client side. The server side has a lot less code because it only worries about taking the requests and putting them into redis and then sending them back out. The client has the all the display logic. Let’s get started.

Socket.io

The great thing about socket.io is that the client side is just as easy as the server side. The same three basic actions are there, connect, on, and emit. The basic idea is that one side emits to the on event of the other side. Socket.io will keep track of the connection for you. On the the side that listens you get whatever you passed through the connection. Continue reading “node.js, socket.io, and redis: Beginners Tutorial – Client side”

node.js, socket.io, and redis: Beginners Tutorial – Server side

Download the src(github)
View the demo

This project has been updated since this was published. The project now includes Express 4.2, Socket.IO 1.2, newer version of Redis, promises, and testing. It is all around better and more up to date. You can read the new post here.

This is a simple application that I have used to try my hand at node.js, socket.io, and redis. The idea came from a co-worker who asked me if I could build an application to collect the votes of our other co-workers to determine where to eat. An initial inspiration came from geoloqi through a game they created. It introduced me to using node.js, socket.io, and redis together to keep multiple browsers in sync.

I had a period of rethinking the entire app design as node.js and redis are both new and different concepts. Node.js requires you to think about how and when(<- this is important) each statement will run. Fenn Bailey has a great post about how ifs and loops work completely different in node (or not at all depending on your point of view). You can get a lot of nulls out of loops if you do not plan out correctly. Redis is a simple and fast key-value store. It has some useful built-in types, but what it doesn’t have is a schema. You cannot map out a row with an id and multiple fields. You have to map your data to keys and sets. I will cover all of my node and redis issues and solutions in this post.

node.js

I am not really doing a lot with node.js here. Node is filling in the server role to give me a platform that I can build from. Although technically that’s what every node.js application does. My node.js parts just listen on 8080 and returns “Hello Socket” on any request. I could have pulled in the HTML to serve from node, but I am just letting Apache serve it. All the good parts are in socket.io which connects through javascript.

One thing you will run into is that node doesn’t have a daemon. When you are debugging this is not an issue as you just run it from the command line and let go. You can run the process in the background, but that is prone to issues. I have found forever as a great way to keep a node.js app running.

The only ‘true node.js’ code in my app.js:

var app = require('http').createServer(handler)
  , io = require('socket.io').listen(app, {origins: '*:*'});

var redis = require("redis");
client = redis.createClient();

app.listen(8080);

function handler (req, res) {
	//I don't know if I need this
    res.writeHead(200);
    res.end("Hello Socket");
  
}

Redis

I am first going to cover redis and my data design and then move on to socket.io. Redis is a self described “open source, advanced key-value store”. This is a very different concept from relational databases. The easiest way to think about is that in a RDBS you create rows which can have mulitple fields, compared to redis where each row is a field. For example in a RDBS if you want to hold the URL for an image you would add a text/varchar field in your row. In redis you would create a user:userid:img key which would hold the data. The use of no schema key value store was one reason I wanted to use redis, as I have never used something like this to store application data. Another reason was speed. Redis is an in memory store. Simple value calls are fast. This application does not have a lot of data and that is a good use case for redis. If I had bigger datasets I would probably use something else.

Data Structure

The data structure really just breaks down to users and votes. Each user is part of an area (the area is set on the client side by adding a hash to the URL) and gets a vote in each area. That’s it. Here is what a user would look like if you collected all the keys.

  • id – [area]:users:[username]
  • image URL – [area]:users:[username]:img
  • vote – [area]:users:[username]:vote

The id is stored in a set with a key of [area]:users. If given an area you would be able to list all the users including an image, and that users vote. The id is also stored in a another set that tracks the votes with a key of [area]:votes. Again if given an area you can list all the votes along with the user that gave the vote. That is it, the entire data structure.

I also use the expiration method of redis to expire all the keys. Each value is set to expire two hours later. This allows the application to reuse areas for different meals (lunch and dinner). It also stops the store from growing too large. This works great for all the user values. It does not work so great for sets. When I try to expire a set, redis turns the set into a value key. I would only have one of the values in the set and it would overwrite the current value when I tried to set anything to it. This is not what I wanted.

I will note that I am using redis 1.2 which is the version in the default repos of my version of Ubuntu. I know that I can download and compile a new version or use a ppa, but I did find a way to make it work how I wanted.

To ensure that the sets did not live on forever is that I created another key called timer for each area ([area]:votes:timer) and set an expiration on that. That key was then put into a set called expireKeys that is looped through every 10 minutes to see if the timers have expired and then delete the matching sets. It is not a lot of overhead as it only runs every ten minutes and it makes sure that the redis store stays clean.

Let’s look at some code on how to save a user to redis in node.js. The client variable is created and set earlier in the script as var client = redis.createClient();

function setUser(username, img, area, expire){
	client.set(area+':users:' + username, username);
	client.expire(area+':users:' + username, expire);
	client.set(area+':users:' + username + ':img', img);
	client.expire(area+':users:' + username + ':img', expire);
	//set a timer
	client.set(area+':users:timer', expire);
	client.expire(area+':users:timer', expire);
	client.sadd(area+':users', area+':users:' + username);
	//add the set to the expire set
	client.sadd('expireKeys', area+':users');
};

This is pulled out into a function and is called when a user addition event is fired. It is very straightforward, key is created with set and then expired with expire. The identity (area:users:username) is then added to the set area:users with sadd.

Setting a vote is very similar.

function setVote(username, area, fs, expire){
	client.set(area+':users:' + username + ':vote', JSON.stringify(fs));
	client.sadd(area+':votes', area+':users:' + username);
	//add the set to the expire set
	client.sadd('expireKeys', area+':votes');
	//set a timer
	client.set(area+':votes:timer', expire);
	client.expire(area+':votes:timer', expire);
};

There is not history with this implementation. If you voted before your vote is overwritten in the set as each voter can only be in the set once. The same is true with users. This does open up impersonation and vote tampering, but with this application I view that as more of a people problem than a technical problem. I also do not think it is a serious issue as all votes and users are gone in a couple of hours.

I will cover the function that checks for expired keys now. This demonstrates how things flow in node.js.

function checkExpires(){
	//grab the expire set
	client.smembers('expireKeys', function(err, keys){
		if(keys != null){
			keys.forEach(function(key){
				console.log(key);
				client.get(key+':timer', function(err, timer){
					//grab the timer
					if(timer != null){
						//timer exists check the ttl on it
						client.ttl(key+':timer', function(err, ttl){
							//the ttl is two hours and if it is under an hour
							//and a half we delete it
							if(ttl < 5400){
								console.log(ttl);
								client.del(key);
								client.srem('expireKeys', key);
							}
						});
					}else{
						//the timer is gone delete the key
						client.del(key);
						client.srem('expireKeys', key);
					}
				})
			});
		}
	});
};

There are a lot of brackets. This is because in node everything is done in a callback. You make a request or in this case a redis call and the value is returned in a callback. You do not write var ttl = client.ttl(key+’:timer’);, you write client.ttl(key+’:timer’, function(err, ttl){//use ttl here});. In the function above all the tabs and brackets are there because it is running a callback inside of a callback inside of a callback. When we first we get the keys, this will be returned as an array. If we try to set a variable to this in the current scope it will always return null. This is different from writing javascript on the client side. The data you want only lives in the callback, so everything happens in the callback. Callbacks are not anything new to javascript and if you can adjust to this thinking, writing node apps becomes a whole lot easier.

Socket.io

Socket.io makes it trivial to connect multiple browsers and have them react in realtime with each other. There are many chat demos out there that show how to take a message from one browser and send it to all the others immediately. That is all this application is doing. When someone votes it is broadcast out to everyone else in the same area. In the context of node.js, socket.io is where all the events are created and handled. Socket.io has all the wiring to respond to the clients requests.

First thing to do is listen for connections. Because this is node everything else happens in the callback from this. This application is listen on the /users path.

var users = io.of('/users').on('connection', function (socket) { //everything here });

We get a socket variable out of this that should be our connection to the client. We now listen for specific events from the client. Socket.io makes this really easy. Emits send a message and ‘on’ catches the message. In this application we have three events we created and one built-in event; add, addVote, getVotes, and disconnect. Socket.io also has support for rooms. This allows you to group your connections into specific, um… well, groups. You can then send a broadcast to just connections in that room. Just like everything else socket.io does this is very easy.

Here is the add event.

	socket.on('add', function(username, img, area){
	if(!usernameExists(area, username)){
		setUser(username, img, area, 7200);
	}
		//var test = new User(username, img, area, socket.id);
		socket.join(area);
		user = new User(username, img, area, socket.id);
		socket.set('user', user);

		});

The socket.on method is where you specify the event name and what parameters it is expecting. In this method we store the username and img URL in redis with the setUser function and then socket.join the area which was specified.

Here is addVote.

	socket.on('addVote', function(fs){
		if(user !== null){
				setVote(user.username, user.area, fs, 7200);
				io.of('/users').in(user.area).emit('vote', {username: user.username, img: user.img, fs: fs});
				//socket.emit('vote', {username: user.username, img: user.img, fs: fs});
		}
	});

The fs parameter is a FourSquare venue object. We store the vote in redis with the current users information (which was set in the add event). We then broadcast it to everyone in the current area. With socket.io it is one line of code. This will emit an event to everyone that has added a user in this area.

Here is getVotes.

	socket.on('getVotes', function(){
		var area = user.area;
			client.smembers(area+':votes', function(err, votes){
				if(votes != null){
					votes.forEach(function(key){
						client.get(key, function(err, username){
							client.get(key + ':vote', function(err, vote){
								client.get(key + ':img', function(err, img){
									socket.emit('vote', {username: username, img: img, fs: JSON.parse(vote)});
								});
							});
						});
					});
				}
			});
	});

Notice all the brackets. We are calling callbacks inside of callbacks. It will grab one bit of info (for example the img of the user that made the vote) and send it to a callback. When we get 5 callbacks in we finally emit the info back to the asking client. My first iteration through when writing this I was trying to populate an array with all this info and make one emit call. I kept getting nulls everytime. I finally quit trying to fight node.js and work with how it was architected by using callbacks.

Summary

I am sure there are better ways to do this, but it does work. This application has helped me think differently about how to design an app. Key value stores and evented I/O is something I have never worked with before. Hopefully I can jumpstart someone down these paths by solving a problem. I will cover the client side in the next post.

Facebook SDK Login for Zend Framework Tutorial part 2

Download the src(github)
View the demo

This is the second part of using the Facebook SDK as a login mechanism. The first covered how to setup each login button and how that interacted with javascript. This post will cover how the auth adapters work with the controller to verify credentials. At this point you should know how to setup a button in the application.ini.

AuthController

The controller is called from a route. The path /oauth maps to the oauthAction and /ajax maps to the ajaxAction. The main difference between these two actions is how they respond. Oauth redirects the page and ajax responds in JSON.

init()

The init() function is run every time the controller loads before any actions. This is a great place to put code that is shared between all actions. Continue reading “Facebook SDK Login for Zend Framework Tutorial part 2”

Facebook SDK Login for Zend Framework Tutorial

Download the src(github)
View the demo

I previously had written about using the Facebook SDK inside of Zend Framework. It did not go very deep into actually using the Facebook SDK. In this post I will show you how to use it as a login system. But wait there is more. I will also show you how to use Twitter, Google, and essentially any other service you want. I use the abstraction that Zend Framework gives you for authentication and add one more layer to that so that you can easily plug in new services to authenticate to.

Zend Framework Authentication
A successful login.

Zend Framework Setup

This is a basic Zend Framework 1.11 install. I did make a few modifications to it. Because one of the authentication methods is OpenId using Google you have to patch the Zend_Openid_Consumer. You can see this why in the Zend Framework issue browser. The other is a way to read attributes from OpenID. This was written by Chris Bisnett. That is the only modifications to the core of Zend Framework. You will want to make sure that you have the most current version of the Facebook SDK. The included Facebook SDK is 3.1.1.

High Level Overview

I will first walk you through how it authenticates any provider and then specifically discuss each that I have. Basically this is what the application does: Continue reading “Facebook SDK Login for Zend Framework Tutorial”

Google Event Tracking

I will talk about event tracking in Google Analytics today. Almost everyone has heard of Google Analytics and I am guessing you are currently using it to track page views. All you have to do is pop a little javascript on your page and you have hundreds of ways to view the traffic your site receives. If you are not using event tracking though, you are missing out on quite a few data points. For a quick primer look at Google’s documentation.

Events

I won’t go into the general idea of tracking events. You have to decide what constitutes an ‘event’ that you want to track. A quick search turns of thousands of blogs that will give you ideas. I want to specifically talk about specific use cases. The first is a one page javascript application. When you are only tracking page views you will miss 90% of the interaction on the page. You have to piggyback on events to see what is happening in the app. The other case is errors. Javascript is client side so if it breaks you do not see it. Event tracking can track these for you.

App Interactions

I will use RunBrowser as my example app. It is a single HTML page javascript app. I used the event listeners on buttons to log events to Google. I did this inside of my AppController object. You can specify the category, action, label, and value for each event. I mainly just used category and action as it was a small app. I tied the main Start button to a Runbrowse category and Start action. Then I tied the save or clear button presses to the Runbrowse category and Save or Clear actions respectively. At this point I can view how many people started runs and then eventually saved them. This data would be lost without event tracking.

Here is a report called Events Flow:
Events Flow

You see that 35 people from the US viewed the page during the selected dates. 4 of them viewed a saved run and 31 started a new run. 20 of those 31 paused it to hit clear or save, I did not extend the image past that, but you would be able to see how many people eventually saved their run (not the actual run and the data associated just the event of saving it). Out the 11 that did not pause 3 had errors. Without event tracking I would have only seen 35 visits from the US and missed out on all this other data.

Errors!

Tracking errors with events is a great way to see what is happening out in the wild. As we saw in the events flow report I had some errors come up. In my error handler for GPS I track the category as Error and the action as the error code. The error codes match what is returned from the location object according to the API. The three values map to 1= PERMISSION_DENIED, 2 = POSITION_UNAVAILABLE, and 3 = TIMEOUT. I can view my Error category and get a rundown of all the errors on my page.
Errors

This report shows I had 12 events where permission was not given to track the location and 4 timeouts. The other event is the clear modal after the error. You could have client side issues and errors and most people will not send a bug report back to you. With event tracking of errors you can see exactly how many errors and hopefully why. You can also tie in other information like browser and OS version to see if you have a specific bug that only affects a subset of users.

Unobtrusive Event Tracking

I found a great jQuery plugin to do Google Event tracking. Because it is jQuery you don’t have to worry about cross browser issues. It allows you to easily add event tracking to an existing page or to on a new page. It attaches right to the click event handler so you don’t have to muddy up your HTML with a lot of onclick attributes. It has the ability to use HTML5 data-* attributes to set category, action, label, and value. It has many options that should be able to suit your needs. It is also hosted on GitHub.

HTML 5 run tracking application

Download the src(github)
View the demo

The github repo has been updated. Some of the code will not match what is currently in the repo.

An idea that I have had for awhile is a RunKeeper like application that runs completely in the browser. This is not due to any inherit issues with the RunKeeper application, just more of a proof-of-concept. HTML 5 makes this easy. We can save old runs and run without an Internet connection making it almost a native app.

The only phone I have tested this with is an iPhone 4S as that is the only phone I have access to. It should work with an iPhone 4 and possibly a 3GS, but I cannot say for sure. Android should work as well as it should support everything I am doing with HTML 5 and javascript. Desktops can also try it out if you are using any modern browser: Chrome, FireFox, or Safari. You won’t get a path and possibly get an error if your browser cannot get a location. I have tested it in IE and it does not work. I have no plans to try and make it work as the only place I was trying to make it work was the iPhone. Although not supporting IE has made the code a lot cleaner.

RunKeeper in the browser: RunBrowser

HTML 5 – almost a native app

HTML 5 gives us many new methods to make a webpage an application. Before HTML 5 there were many workarounds to make a webpage do what we wanted it to. Now they are built right into the browser. The first thing I am going to cover is application cache. Continue reading “HTML 5 run tracking application”