Featured

Build Complex Express Sites with Redis and Socket.io

I have another video series ready from Packt Publishing named Build Complex Express Sites with Redis and Socket.io. This course builds on the foundation from the other course I built for Packt Publishing earlier this year, The Complete Guide to Node.js.

It starts off by covering what and why of Redis. Then goes to building Express sites using Redis. After that, the course jumps to using Socket.io to build real-time sites. Just like Redis the course begins simple and covers what Socket.io is. The course finishes up building sites in Express utilizing Socket.io.

Featured

The Complete Guide to Node.js Video Series

I have just recently finished creating a video series for Packt Publishing named The Complete Guide to Node.js. The course focuses on taking someone completely new to Node.js and giving them a great foundation to start building real applications. There is a good mix of theory and practical application as the viewer goes through the course. First, an idea is introduced and explained. Then the idea is applied.

While I was waiting on Packt to send me some finalized videos as a preview, I received a twitter endorsement for the course.

Enough talking about the course, here are two videos that demonstrate the content of the course. Enjoy!


Featured

Book Preview (Building Scalable Apps with Redis and Node.js)

Download the src(github). This is all of the code from the book.

I am excited to present an excerpt from my book, Building Scalable Apps with Redis and Node.js. It is from the second chapter entitled Extending Our Development with Socket.IO. In the first chapter we created a simple Express app and now we are rolling in Socket.IO. This is only the end of the chapter. I, of course, highly recommend the book and hopefully after reading this you will agree.

This has been adapted from the source to better fit my blog’s presentation and has a few typos fixed, which means that there are a few typos in the book :(.

Using Socket.IO and Express together

We previously created an Express application. This application is just the foundation. We are going to add features until it is a fully usable app. We currently can serve web pages and respond to HTTP, but now we want to add real-time communication. It’s very fortunate that we just spent most of this chapter learning about Socket.IO; it does just that! Let’s see how we are going to integrate Socket.IO with an Express application.

We are going to use Express and Socket.IO side by side. As I mentioned before, Socket.IO does not use HTTP like a web application. It is event based, not request based. This means that Socket.IO will not interfere with Express routes that we have set up, and that’s a good thing. The bad thing is that we will not have access to all the middleware that we set up for Express in Socket.IO. There are some frameworks that combine these two, but it still has to convert the request from Express into something that Socket.IO can use. I am not trying to knock down these frameworks. They simplify a complex problem and most importantly, they do it well (Sails is a great example of this). Our app, though, is going to keep Socket.IO and Express separated as much as possible with the least number of dependencies. We know that Socket.IO does not need Express, as all our examples have not used Express in any way. This has an added benefit in that we can break off our Socket.IO module and run it as its own application at a future point in time. The other great benefit is that we learn how to do it ourselves.

We need to go into the directory where our Express application is. Make sure that our pacakage.json has all the additional packages for this chapter and run npm install. The first thing we need to do is add our configuration settings.

Adding Socket.IO to the config

We will use the same config file that we created for our Express app. Open up config.js and change the file to what I have done in the following code: Continue reading “Book Preview (Building Scalable Apps with Redis and Node.js)”

Practical Functional JavaScript

I have made a video to go alongside my last post series. In that series, I focused on showing what it means to write practical functional Javascript. Just like all my other posts, this is done by building something, a Mad Libs generator. The problems that arise from building a Mad Libs generator are perfectly suited to solving in a functional manner. The series is broken up into three parts, first creating a functional backend, then rendering it to the page, and finally testing.

This video maps closely with the first post. It firsts introduces functional ideas like composition and currying. Then it jumps right in and shows how to use these in an actual application. Watch and enjoy.

Testing Functional Mad Libs

Download the src(github)
View the demo

This is the final post in a three-part series. The first post dealt with building a functional foundation and the second post was about building a functional UI. This post will be about testing everything.

Testing

Testing is very important, but sometimes it gets left behind. This can be because it is not clear how or even what to test. Tightly coupled code is a testing nightmare. It is very hard to unwind specific units of code to test. In addition to this mocking can become a huge task where you have to recreate all the resources the application needs.

This is where functional design comes in. By definition, it should be easy to test. There will be many functions that should only rely on what is passed into them. Mocking becomes trivial. Deciding what to test becomes trivial. Overall testing becomes trivial.

In this post, I will not bore you with the details about every test. Testing can be repetitive. I will highlight how I tested, how I mocked, and highlight any interesting parts.

How to test

There are three mostly agreed upon ways to test, unit, integration, and end to end. The definitions for these will change depending on who you ask, but they are as follows. Unit tests are built to test one specific function. This means that no dependencies or mocking should be used. These tests are the simplest. Next up is integration testing. At this point, pieces are starting to be put together, integrated if you will. Integration tests will include unit tested pieces and some mocking. Although mocking is not always required. Then the final testing is end to end. This is where the entire application is built and tests are run. The previous tests usually do not need more than a test runner. End to end will involve more tooling.

Does this sound familiar? It should because this is exactly the same approach to writing functionally. First, you write simple functions. Those functions are then combined to create larger pieces. Then everything is put together to create the application. Each step maps to unit, integration, and end to end.

This means we start with the simplest functions and run them through their paces. Because these are simple functions the tests pretty much write themselves. Here is an excerpt from basic_functions_test.js.

const assert = require('assert');
const {filterFunc, addField, filterMadLib} = require('../src/basic_functions.js');

describe('filterFunc test for pos', () => {
  let verb = {pos: {Verb: true}};
  let noun = {pos: {Noun: true}};
  let adjective = {pos: {Adjective: true}};
  let adverb = {pos: {Adverb: true}};
  it('should match each word type', (done) => {
    assert.equal(filterFunc('Verb', verb), true);
    assert.equal(filterFunc('Noun', noun), true);
    assert.equal(filterFunc('Adjective', adjective), true);
    assert.equal(filterFunc('Adverb', adverb), true);
    done();
  });
  it('should not match different word types', (done) => {
    assert.equal(filterFunc('Verb', noun), false);
    assert.equal(filterFunc('Verb', adjective), false);
    assert.equal(filterFunc('Verb', adverb), false);

    assert.equal(filterFunc('Noun', verb), false);
    assert.equal(filterFunc('Noun', adjective), false);
    assert.equal(filterFunc('Noun', adverb), false);

    assert.equal(filterFunc('Adjective', noun), false);
    assert.equal(filterFunc('Adjective', verb), false);
    assert.equal(filterFunc('Adjective', adverb), false);

    assert.equal(filterFunc('Adverb', noun), false);
    assert.equal(filterFunc('Adverb', adjective), false);
    assert.equal(filterFunc('Adverb', verb), false);
    done();
  });
});

The tests are clear and very simple. The function only does one thing so we know what to test.

Integration testing

We can move to the next step, integration testing. The great part about this is that we have already integrated the functions by composing them. We can see an example of this kind of test in render_functions_test.js.

const assert = require('assert');
const {createRenderElements, enterRenderElements, doneRenderElements} = require('../src/render_functions.js');

let text = "Somebody once told me the world is gonna roll me. I aint the sharpest tool in the shed.";

describe('createRenderElements test', () => {
  it('should should have MadLib on the indexed', (done) => {
    let create = createRenderElements([], text);
    let icreate = createRenderElements([1,3,5], text);

    assert.equal(create.length, 19);
    assert.equal(create[0].text, 'Somebody');
    assert.equal(icreate.length, 19);
    assert.equal(icreate[0].text, 'Somebody');
    assert.equal(icreate[0].MadLib, undefined);
    assert.equal(icreate[1].MadLib, true);
    assert.equal(icreate[3].MadLib, true);
    assert.equal(icreate[5].MadLib, true);
    done();
  });
})

This excerpt makes it clear that these tests are just as easy to write as our unit tests. Remember that each one of the *renderElements functions is composed of up to six different functions. Now some of these are Ramda functions, but the point still stands that this is an integrated test of functionality.

Mocking

Mocking can be trivial if we have built our functions correctly. If our functions only do one thing then it is clear what we need to mock for that function. In our application, the only thing we need to mock is the document object. And even that only needs a few things mocked to work. Looking at what methods are used on document we see we only need to add some child manipulation tools (appendChild, removeChild, and firstChild) and some properties. Here is the document mock object, it is only 17 lines total.

function Element(node) {
  this.nodeName = node;
  this.dataset = {};
  this.children = [];
  this.appendChild = (el) => { this.children.push(el);},
  this.removeChild = (el) => { this.children.splice(this.children.indexOf(el), 1)},
  Object.defineProperties(this, {
    'firstChild': {
      'get': () => { return this.children[0]}
    }
  });
}

module.exports.createElement = (node) => {
  return new Element(node);
}

This was easily done because we kept each function simple and we could then compile what we need the mock object to do. Now here is some code to test our IO functions. The document mock object will need to be used here. We are testing if this function will remove all children and then add our elements back.

const assert = require('assert');
const {render, setAttribute} = require('../src/io_functions.js');
const document = require('./document_mock.js');

describe('render test', () => {
  it('should remove all current elements', (done) => {
    let root = document.createElement('root');
    root.children = [1,2,3,4,5];
    render(root, []).run();
    assert.equal(root.children.length, 0);
    done();
  });

  it('should add new elements in', (done) => {
    let root = document.createElement('root');
    root.children = [1,2,3,4,5];
    render(root, [1,2,3]).run();
    assert.equal(root.children.length, 3);
    assert.equal(root.children[0], 1);
    assert.equal(root.children[1], 2);
    assert.equal(root.children[2], 3);
    done();
  });
})

We can now trust that this code will execute the correct methods on the real document object when it is used in the browser. Now let’s look at using it to create specific elements.

We have two functions that create elements to go in the DOM, spanMap and inputMap. To test these functions we need to pass in the document mock object and then inspect some of the properties. These are not wrapped in an IO monad because it is a simple map. One object to another. Creating an element is a pure function and adding them to the DOM is an impure function. Here is the code.

const assert = require('assert');
const document = require('./document_mock.js');
const {spanMap, inputMap} = require('../src/dom_element_map_functions.js');
const getTerms = require('../src/text_functions.js').getTerms;

let terms = getTerms(" dog eats ");
terms[0].Index = 0;
terms[1].Index = 1;
terms[1].MadLib = true;

describe('spanMap test', () => {
  it('should create spans based on term', (done) => {
    let span = spanMap(document, terms[0]);
    let madSpan = spanMap(document, terms[1]);

    assert.equal(span.nodeName, 'span');
    assert.equal(span.className, 'Noun');
    assert.equal(span.dataset.index, 0);
    assert.equal(span.title, 'Noun');
    assert.equal(span.innerHTML, ' dog ');

    assert.equal(madSpan.className, 'Verb PresentTense MadLib');
    assert.equal(madSpan.dataset.index, 1);
    assert.equal(madSpan.title, 'Verb PresentTense');
    assert.equal(madSpan.innerHTML, 'eats ');
    done();
  });
});

First, we create the element, then look at the properties. I want to highlight the fact that we have tested DOM manipulation and element creation without using a headless browser like PhantomJS. The testing was accomplished with a 17 line mock. I make this point because many times I see way too much mocking because of bad design.

End to End Testing

Well, I don’t have any end to end testing. The only file that needs this is the final index.js. The tests needed for that would mainly fall in the category of making sure events were wired up correctly.

Two paragraphs ago I railed against over mocking for integration testing, but things are different when discussing end to end testing. We want to create an environment as close to what the real world will be. Now it is fine to pull in headless browsers to do testing. Why is it wrong for integration testing? In short, it is code design. Why not use a headless browser for unit tests? If the code requires this level of mocking, then it can be simplified. Again there are always exceptions, but we should be creating simple and pure functions from the start.

Summary

We have looked all the different methods of testing. I think the most important point is that functional code is easy to test. The tenets of functional design fit with testing perfectly. Start off with simple functions that can be easily unit tested. Then combine those functions with higher order functions. The results of the combination can then be used in integration testing. The final step is to test everything assembled with end to end testing. It is easy to know what and how to test everything this way.

Functional Front End: Why React and Redux?

Download the src(github)
View the demo

This is the second post of a two-part series. The first post covered building a functional back-end for making Mad Libs. We will focus on the front end. React and Redux are not actually used in the code, but they are used in execution. I will show the reason why React and Redux are so powerful. In addition to this, we will look at using IO monads and put everything together. Let’s get started and pick up where we left off.

So, Why React?

We will start with React as we need to show how to modify the DOM. Here is some simple code that we will discuss.

const R = require('ramda');
const IO = require('monet').IO;

//IO monad stuff
let addChildren = (elements, root) => {
  R.forEach((el) => {
    root.appendChild(el);
  }, elements);
};

module.exports.render = R.curry((root, elements) => {
  return IO(() => {
    while (root.firstChild) {
        root.removeChild(root.firstChild);
    }

  addChildren(elements, root);
  });
});

module.exports.setAttribute = R.curry((attribute, element, value) => {
  return IO(() => {
    element[attribute] = value;
  });
});

To not sidetrack this discussion we will not get into monet and the IO monad right now. First look at render, all it does is delete everything in a certain root element and then add all the elements from an array as children. This is simple and naive, but I am arguing here that this is the real value of React. Continue reading “Functional Front End: Why React and Redux?”

Functional Mad Libs

Download the src(github)
View the demo

I am continuing my path towards functional programming that I have been dabbling with the last year or so. This usually meant I would build things object-oriented for the most part, but then anytime an Array popped up I would transform it using functional patterns. This is pretty easy and does not force me to get uncomfortable. Although it is still very useful as there are many times collections come up. I just finished a great book called Functional Programming in JavaScript which I have reviewed. I learned new ways of writing code and I want to try them out.

Which leads me to the focus of this post. I have built a functional Mad Libs site. This came from a joke filled conversation at work about Mad Libs. The great thing about Mad Libs is that it is a perfect functional problem. There is a list of words. Some of the words need to be replaced. Get new words. Then replace the old with the new. This is an easy computational problem. I wanted to make the entire process as functional as I could. Let’s get started.

This is not an intro to functional programming. Many blog posts have been written and I do not think that I would add much to what has already been said. The purpose of this is to show an application written in a functional manner. If functional programming is new to you, either read Functional Programming in JavaScript, watch the first few videos by Mattias Petter Johansson(funfunfunction), or choose any of the top articles on Medium about Functional Programming.

Ramda

I want to start with Ramda. The Mad Libs site could not have been built without Ramda. It is the most used tool/utility/glue/peanut butter of the site. Ramda is usually pulled in as the variable R and if you look at the code you will see R everywhere. I don’t think I will write JavaScript without Ramda again, it’s that good. Ramda should have the type of ubiquitousness of jQuery. Continue reading “Functional Mad Libs”

I found a functional way to build Express’s middleware

Using functional design is the new cool thing. This is especially true when talking about JavaScript. Functional programming can make complex code much simpler and much shorter. I am going to highlight some code that I wrote for my Packt Publishing video course, The Complete Guide to Node.js.

I want to note first that this is not a perfect apples to apples comparison. It is also not a judgment on the code written for Express. Express was not built to be functional and cannot be faulted for not using functional ideas. I just want to highlight a functional way of accomplishing a similar task.

The code

We are going to specifically look at middleware in Express. We will focus on the implementation of next. Middleware accomplishes the task of taking a bunch of functions that need to run on a request. Some middleware needs to run on every request and others only on specific requests. This means functions that are not the final item in the chain need to continue the chain. This is done with the next function. If you have not used Express middleware I recommend Express’s documentation.

Let’s look at how Express does this on Github. The code we will look at is in lib/router/index.js. We will look at the handle function and focus at the code starting at line 178. Again I will highlight that the Express version does more than just handle the middleware stack.

Here we can see that there are 12 ifs in the while as it loops over each item in the middleware stack. The good thing is that these ifs are not nested, if they were it would be crazy. This makes it a little difficult to really grok what is happening quickly. Go ahead, jump in and see how long it takes you to determine what is happening.

My code

This is not production ready code but it shows a different way of approaching the same problem. Let’s look at the code first and then discuss what is happening.

const url = require('url');
var routes = [];
var registerRoute = (method, url, fn) => {
  routes.push({method: method,
  url: url,
  fn: fn});
};

var routeMatch = (route, url) => {
  return route === url || route === undefined;
};

var methodMatch = (routeMethod, method) => {
  return routeMethod === method || routeMethod === undefined;
};

var isError = (fn) => fn.length === 3;
var isNormal = (fn) => fn.length === 2;

var mapToRouteMatch = (reqUrl, reqMethod) => {
  return (route) => {
    return routeMatch(route.url, reqUrl)
    && methodMatch(route.method, reqMethod);
  }
};

var handleRequest = (req, res) => {
  var matchedRoutes = routes
  .filter((route) => isNormal(route.fn))
  .filter(mapToRouteMatch(url.parse(req.url).pathname, req.method));
  try{
    matchedRoutes.some((route) => route.fn(req, res));
  }catch(e){
    let errorRoutes = routes
    .filter((route) => isError(route.fn))
    .filter(mapToRouteMatch(url.parse(req.url).pathname, req.method));
    errorRoutes.some((route) => route.fn(req, res, e));
  }
};

module.exports.registerRoute = registerRoute;
module.exports.handleRequest = handleRequest;

This is 42 lines that completely implements a routing middleware much like Express. The routes array and registerRoute are the core data of this. In fact, if you wanted you could change registerRoute to use to make it, even more, like Express.

There are then four filtering functions (routeMatch, methodMatch, isError, and isNormal). They are all one line of code so I won’t spend time discussing them. Next there is a higher order function mapToRouteMatch. This takes a URL and a method and then combines the return of the matching functions. This allows us to make routes that match both/either the method and/or the URL. This gives us the flexibility to run a piece of middleware for every request or just one.

One quick aside, the function mapToRouteMatch is really just a partially applied function. The function partially applies the URL and method which returns a new function that will then expect each route. Which the function will get when mapped over the array.

Finally we get to the core handling, handleRequest. Thinking functionally, there is a clear way to get which pieces of middleware to run on this request, filter! We already have functions that can filter down the array to just the functions that have 2 parameters (req and res) and match the current URL and method. After that we just run some over the array. some will continue over each item until one of them returns true. This is perfect if any function is the final function in the chain, just return true.

This is wrapped in a try/catch. If there is an error we catch it and then find each error middleware and execute it with the error. Let’s see how to actually use this now.

Here is a simple server with two endpoints and six total middleware.

const http = require('http'),
      routes = require('./routes.js');

//actual responses
var log = (req, res) => {
  console.log(`${req.method} ${req.url}`);
  return false;
};

var poweredBy = (req, res) => {
  res.setHeader('X-Powered-By', 'ejosh.co/de');
  return false;
};

var index = (req, res) => {
  res.write("<html><head><title>Page" +
"</title><head><body><h1>Our Web Application</h1>" +
"</body></html>");
  res.end();
  return true;
};

var createError = (req, res) => {
  throw new Error('this will always throw');
  return false;
}

var defaultRoute = (req, res) => {
  res.end();
  return true;
};

var errorRoute = (req, res, err) => {
  res.write(err.message);
  res.end();
  return true;
}

routes.registerRoute(undefined, undefined, log);
routes.registerRoute(undefined, undefined, poweredBy);
routes.registerRoute('GET', '/', index);
routes.registerRoute('GET', '/error', createError);
routes.registerRoute(undefined, undefined, errorRoute);
routes.registerRoute(undefined, undefined, defaultRoute);

var server = http.createServer();
server.on('request', routes.handleRequest);
server.listen(8081, '127.0.0.1');

We are using the built-in HTTP server and including the code we just looked at as routes.js. The next six functions should look really familiar if you have ever built an Express middleware. The main difference is that there is no next function. Just return false to continue processing and true to stop processing.

Next is the section where the routes are registered. This is different than Express as it is much more explicit. Every call has to pass all three parameters; method, URL, and function. Passing in undefined means that it will match for every request. Remember order matters.

Finally, it is wired up by starting the HTTP server and setting handleRequest to handle the requests.

As you can see we now have a functioning Express-like router and application in just 90 lines of code. The router does not have all of the features of Express, but hopefully, it is clear where the features can easily be added. For example regular expressions could be added as another function and used in routeMatch. Ultimately the main advantage is that we have simplified finding and running the correct middleware down to two filters and then a map (some is essentially a map). This allows us to use very simple functions for the actual logic portion.

The Secret of Functional Programming in JavaScript

I just finished an amazing book by Luis Atencio named Functional Programming in JavaScript. It is published by Manning and you can purchase it from Manning.

If you have been reading my blog you will see that I have been trying to push myself further into the functional paradigm. I have been classically trained as an object oriented programmer, much like most of the programming world. The idea of functional programming initially just seemed weird and something that people talked about, but never really implemented. Although I was drawn to the compactness of code and ease in which someone can reason about what is happening.

I purchased this book as a MEAP (Manning Early Access Program). MEAP means I received the eBooks as they were written, but I have found it is difficult to finish technical books in a digital format. I still prefer the physical book. The book has been out since the summer, but I have just now finished it. I highly recommend it to anyone wanting to learn more about functional programming.

The book starts off with the common cases for why use functional programming. It is similar to all the other articles on this subject. I would not count that has a detraction as Luis makes a good case for why functional programming.

Next, the book really becomes great in the next part, Get Functional. It takes a simple problem and shows the difference between building an imperative version and a functional version. This part finishes with one of the best descriptions of a monad I have ever read. Monads are one of those things that there are a lot of poor explanations of out on the Internet. I may one day add to that pile. There is a wonderful Douglas Crockford quote about monads.

In addition to it being useful, it is also cursed and the curse of the monad is that once you get the epiphany, once you understand – “oh that’s what it is” – you lose the ability to explain it to anybody.
Douglas Crockford

Luis, though, does not fall to this curse and nails both theory and application.

The final part of the book deals with performance and functional reactive programming. The book notes, correctly, that the value of using functional programming is not in any performance gained, but in writing clean easy to understand code. I feel Luis does a good job of wrapping up the core concept of the book and leaving the reader with somewhere to go.

I currently have my next book already purchased and sitting on my desk, Functional Reactive Programming from Manning. This is directly related as it is one of the books recommended in Functional Programming in JavaScript.

Trecco – my First iOS app

I have not posted in awhile, but I have been busy. I find that this is a recurring theme with my blog. I make a string of posts on time and then I get caught up writing code. Back to the post at hand.

Over the holidays I spent some time to learn Swift and write an iOS app. Trecco was the app that I built. It will record a voice note, use IBM’s Watson to transcribe the voice note, and then save it to a Trello board as a card. The name Trecco comes from recording a note and saving it to Trello.

Trecco App on iPhone

I do recommend that you download and use it. Especially if you use Trello.  Furthermore I am using this opportunity to create a new book! While I do not have a title yet, the book will focus on building a Swift iOS app from the ground up. This means starting from an empty folder all the way through submitting it through the app store. Every part will be covered.

I am hoping to release a sample chapter and video, so you can get a feel for what the book will be like.

12 Days of Posts: Day 12

Technically this one is late. I did not account for all the things I would have to do on Christmas eve. Picking a subject and writing around 200 words on it every day for eleven days was more difficult than I expected. I had chosen some subjects before, but 3 were pretty much day of selections.

I wanted these posts to be about the things that I learned or focused on for 2015. I think the main thing I tried to do this year was focus on programming more functionally. The more I learn about functional programming the more I like the way it makes you think. This is especially true from a testing point of view. When writing code I can easily create a lot of little edge cases and then you have to write tests for all of them. If I change how I think about the problem I can usually cut through all of that and make a simpler codebase. I think functional programming does that.

The other thing I have embraced is Docker. I love the ability to have my application running exactly the same in development as production. In my mind this is another simplification of development like functional programming. Once I have it working exactly how I want it in development there is really no work to get it working anywhere else.

This last year has been about writing and developing simpler software. I am not even close to where I think I can be. In 2016 I plan on continuing this. Hopefully you will come with me as well as you read my blog!