uncategorized

Running Jasmine Tests Serially

Jasmine runs test files in parallel for performance reasons. In some cases, however, you may wish to run a series of tests sequentially.
For example, you may have multiple tests that open a port for testing sockets and you will errors if you try to open the port while another
test is using it. You could, put all those tests in one file and run them sequentially as separate tasks but that is really hard to maintain and gets ugly
as your project grows.

I decided to write a module to run tests sequentially. My first thought was. “That should be easy just wrap the tests in promises and use the jasmine api to run the test”.
Unfortunately there is a deficiency in the api and if you call jasmine.execute multiple times you get a real mess. It does not clean up after
it executes and subsequent tests are polluted by data from earlier tests. In order to overcome this, I forked a new process for each test. That clearly won’t scale to thousands of tests but usually you wouldn’t need to run everything sequentailly so it seemed ok for common use cases.

##Requirements for the module

  1. Run test files in sequence
  2. Allow globs for file specification
  3. Run from command line or programatically
  4. Collect data from each test file for later use (e.g. building a healh status dashboard)
  5. Be able to use custom reporters.

##Process flow

  1. Create an array of promises. One promise for each file
  2. Run the promises in sequence.
  3. Report when all promises have resolved.

Each promise forks an instance of jasmine and calls jasmine.execute for a single file and resolves when jasmine exits.

##Running promises in sequence

I have a utility for running promises in sequence serialPromiseRunner

1
serialRunner.prototype.runTasks = function (aTasks) {
  var self = this;
  if(! aTasks){
    aTasks=self.getTasks();
  }
  return new Promise(function (resolve, reject) {
    function *gen(aTasks) {
      for (let task of aTasks) {
        task.args.push(self.results);
        let res = yield task.funct.apply(null, task.args);
        self.results[task.name] = {success: true, results: res};
      }
    }
    Promise.coroutine(gen)(self.getTasks())
      .then(function (results) {
        resolve(self.results);
      })
      .catch(function (err) {
        reject(err);
      })
  })
}

This uses the bluebird promise library and, in particular, Promise.corountine for handling promises using generators
A given task takes the form:

1
{
	name : test,
	args : [test, theReporter, saveData],
	funct: require("./lib/runFork")
}

In this application test is the name of the test file, theReporter is the file location of the jasmine reporter,
and funct is a function which is invoked when the test is to be run. It is
invoked with the arguments provided in args as you can see in the promise runner. That runner also include the results from promises already run so you have those results available but that is irrelevant in this case.

runFork is where the heavy lifting is done:

1
var Promise = require("bluebird"),
	fs = Promise.promisifyAll(require("fs")),
	child_process = require("child_process"),
	util = require("util"),
	EventEmitter = require("events").EventEmitter;
module.exports = (file,theReporter,saveData)=> new Promise((resolve, reject)=> {
	 saveData=saveData?"TRUE":"FALSE";
	"use strict";
	let results = {
		jasmineStarted: [],
		suiteStarted  : new Map(),
		specStarted   : new Map(),
		specDone      : new Map(),
		suiteDone     : new Map(),
		jasmineDone   : []

	}
	let child = child_process.fork(__dirname+"/runJasmine", [ file,theReporter,saveData], {silent: false});
	child.on("exit", (signal)=>{
			resolve(results)}
	);
	child.on("message", (messageObj)=>{
		switch (messageObj.type){
			case "jasmineStarted":{
				results.jasmineStarted.push(messageObj.message);
				break;
			}
			case "suiteStarted": {
					results.suiteStarted.set(messageObj.message.id, messageObj.message);
				break;
				}case "specStarted": {
					results.specStarted.set(messageObj.message.id, messageObj.message);
				break;
				}case "specDone": {
					results.specDone.set(messageObj.message.id, messageObj.message);
				break;
				}case "suiteDone": {
					results.suiteDone.set(messageObj.message.id, messageObj.message);
				break;
				}
		}
	});
	child.on("error",(error)=>console.log(error));
})

It spawns a new child_process runJasmine and listens to that process for messages. In node a child_process is an EventEmitter and we use that to communicate back to the parent. When it receives a message from the child it appends those results to the results hash and when the child exits, we resolve the promise with the results hash. The messaging in the child process is handled by a custom jasmine reporter which we will see in a moment.

runJasmine is very simple. It instantiates jasmine,adds a reporter(s) and invokes jasmine.execute

1
var util = require("util"),
	Jasmine = require("jasmine");
function runJasmine(file, theReporter, saveData) {
	"use strict";
	let jasmine = new Jasmine();
	if (theReporter && theReporter !== 'null' && theReporter !== 'undefined') {
		let rep = require(theReporter);
		if (typeof rep === 'function') {
			jasmine.addReporter(rep());
		} else {
			jasmine.addReporter(rep);
		}

	}
	if (saveData === 'TRUE') {
		jasmine.addReporter(require(__dirname + "/serialReporter")());
	}
	jasmine.execute([file]);

}
module.exports = runJasmine.apply(null, process.argv.splice(2));

When jasmine execute runs it emits messages as it progresses. We use a custom reporter to intercept those messages and send them to the parent process. Our custom reporter does not send any information to stdout.

1
'use strict';
module.exports = ()=> {
	return {
		jasmineStarted: (suiteInfo)=> {
			process.send({type: "jasmineStarted", message: suiteInfo});
		},
		suiteStarted  : (result)=> {
			process.send({type: "suiteStarted", message: result});
		},
		specStarted   : (result)=> {
			process.send({type: "specStarted", message: result});
		},
		specDone      : (result)=> {
			process.send({type: "specDone", message: result});
		},
		suiteDone     : (result)=> {
			process.send({type: "suiteDone", message: result});
		},
		jasmineDone   : ()=> {
			process.send({type: "jasmineDone", message: "Finished"});
		}

	}
}

A jasmine reporter is just an object with methods that jasmine calls at the appropriate time. Normally you would just process that data and emit something to the console(process.stdout) but we handle reporting later. For now we just capture the data and send it to the parent process.

The entry point for the process is runTask which ties all the above pieces together.

1
// a bunch of initialization here (not shown)
//create the tasks from the array of files(theTasks)
theTasks.forEach((test)=> {
		tasks.push({
			name : test,
			args : [test, theReporter, saveData],
			funct: require("./lib/runFork")
		});
	});
	// instantiate the serial promise runner and run the tasks
	let spr = new SPR(tasks);
	spr.runTasks()
		.then((res)=> {
			results = res;
			// we are done. If user has not provided a reporter we print
			// a simple report
			if(saveData && !theReporter){
				writeResults(results)
			}
			//resolve our promise to the calling program with the data
			//we collected
			
			resolve(res);
		})
		.catch(reject)
		.finally(()=>spr = void(0));
})

##Typical output from command line

Using jasmine’s default reporter
Jasmine default
Using our reporter
serial-jasmine

This was an interesting task. I wound up learning a bit about jasmine and how to work with child processes.
If you would like to use the module it is available on npm as serial-jasmine

Share