uncategorized

Leveraging AWS S3 and nconf for Application Configuration

We try to write applications to be context agnostic and use configuration files and environment variables supply context.
The config files are generated at build time using a skeleton template that is fleshed out for the environment being built(local, dev, staging, production).
As we move to a Docker environment this pattern doesn’t work so well because we want a docker image to be usable in all environments, so cannot attach config files to it.
I am sure there are ways to allow docker do this but there is a simpler solution.
If we store the configuration files in an S3 bucket (different files for each environment) and then use an environment variable at run time to determine the appropriate file, the process is very simple.
We leverage nconf to store configuration. On application start we populate nconf and the rest of the applicaion retrieves context from nconf so it does not care (or know) the actual source of the context.

Lets look at a simple example:

1
"use strict";
var AWS = require('aws-sdk');
var Promise = require("bluebird")
var s3 = new Promise.promisifyAll(new AWS.S3()); //I prefer promises to callbacks
var util = require("util");
var nconf = require("nconf");
nconf.env().argv();   //store environment variables and command line arguments
//use defaults for testing. These will not be used if present in argv or env
nconf.defaults({
	BUCKET    : "dvawterconfig",
	CONFIGFILE: "cryptoconfig/common_config.json"
})
AWS.config.update({
	accessKeyId    : nconf.get('AWS_ACCESS_KEY_ID'),
	secretAccessKey: nconf.get('AWS_SECRET_ACCESS_KEY'),
	region         : nconf.get('AWS_REGION')
});
//The deploy process provides BUCKET and CONFIGFILE based on environment either as an envar or command line argument
s3.getObjectAsync({
		Bucket: nconf.get("BUCKET"),
		Key   : nconf.get("CONFIGFILE")
	})
	.then((data)=> { // The data comes back as a Buffer so convert to string and parse since it is a json file
		let obj = JSON.parse(data.Body.toString());
		//add the configuration to nconf
		nconf.add('config', {type: 'literal', store: obj});
		console.log(util.inspect(nconf.get("loggers"), {depth: null}));
	})
	.catch((err)=>console.log(err, err.stack));

Typical output:

1
/Users/donvawter/.nvm/versions/node/v4.2.1/bin/node --debug-brk=52868 --nolazy testbucket.js
Debugger listening on port 52868
[ { logtype: 'bunyan',
    enabled: true,
    streams: 
     [ { level: 'info', stream: 'process.stdout' },
       { level: 'info', path: './logs/cryptogram-info.log' },
       { level: 'error', path: './logs/cryptogram-error.log' },
       { level: 'fatal', path: './logs/cryptogram-fatal.log' } ] },
  { logtype: 'email',
    enabled: true,
    from: 'donvawter@mac.com',
    to: 'donvawter@mac.com' },
  { logtype: 'loggly', enabled: false, tags: [ 'cryptogram' ] } ]

You could, in principle, store all the configuration in environment variables but it would be a painful process to create environment variables for complicated configurations like loggers which are better handled as objects.

The approach is flexible and you can still have the build process create the config files and upload them to the S3 bucket if they are not static. That usually is not necessary because they seldom change.

There is no reliance on Docker so your applications can run locally also without modification.

This is just one example of how, even if difficult for us oldtimers, thinking about files as objects instead of files on a file system makes processes simpler.

Share