uncategorized

Tweaking DNS Update with Nginx and Consul

In an earlier post Dynamic DNS I discussed updating DNS records on CloudFlare with Nginx and Consul.
In that version we just found a matching record at CloudFlare and replaced it with one pointing to the current instance of Nginx. That works fine as long as you only want one A record for a site. If, however, you have you multiple records and use DNS round robbin that approach won’t work.

For redundancy you may also have the site hosted in another data center or even with a different provider. We need a scheme which leaves those records intact.

Requirements

  1. Leave records intact which are functioning.
  2. Remove from DNS any records which point to dead ip’s.
  3. Add a record pointing to the current NginxIp if not present.
  4. When we stop a website (and are going to rebuild, likely with a new ip) remove the DNS record.

Checking the health of existing ip’s.

Since those ip’s are not controlled by the existing Nginx instance or consul we cannot use Consul’s health checks.

In order to check the “health” we make a call to the ip. We append the service name to the path so that if the site is also controlled by nginx and consul it will test the service we are updating. We really don’t care what the response status is as long as the request doesn’t timeout or get refused. If the other site is not controlled by consul then adding the service to the path will likely return a 404 status but that is ok. This isn’t perfect because if the site is behind nginx, but not configured with consul, we are only testing the health of nginx. That seems like a corner case to me. I am not likely to host a site in different regions using different architecures.

We just use Node’s http module for the request. Of course we cannot send by domain name because who knows which host we will actually hit. By appending the service to the path we will hit the internal routing of nginx (assuming we have the same architectue).

1
let checkIfUp = (ip, name)=>new Promise((resolve, reject)=> {
	let http = require("http");
	let options = {
		host: ip,
		path: '/' + name + '/'
	};

	let callback = (response)=> {
		var str = '';
		response.on('data', (chunk)=> { //we don't care about this, really
			str += chunk;
		});
		response.on('end',function () {
			resolve(true)
		});
		response.on('error',function () {
			resolve(false)
		});
	}
	let request = http.request(options, callback);
	request.setTimeout(5000, ()=> {
		resolve(false);
	});
	request.on("error", (err)=> {
		resolve(false);
	});
	request.end();
});

Notice that we set a timeout of 5 seconds, which is rather arbitrary. If the promise rejects we delete the record.

1
let id = record.id;
process.stdout.write('Deleting record for record ' + recordName + ' at ip ' + record.content);
let deletePromise = api.zoneDNSRecordDestroy(zoneid, id)
 .then((results)=> {
		console.log(results);
	})
 .catch((err)=> {
	console.log(err);
   });
});

Cleaning up when we stop a container

This will only work if nginx is still running. If it has crashed and you rebuild then the records will fail the health test above and be removed at that time.
The only reason to do this is good manners. Well not really; The DNS records would remain out there until you rebuilt. If you were scaling down you would never catch them.

1
/usr/local/bin/docker exec ${PREFIX}_nginx_1 /usr/local/bin/node /application/index.js --task deletedns

We just use docker exec to hit the node code on the nginx machine. The delete code is as above and you call it for any record that has the current nginx ip. After that command finishes we stop the container.

Summary

This approach allows us to not interfere with versions of the site hosted in another location and has the advantage of cleaing out dead records.

Share