A few weeks ago I wrote about the theory of AWS Availability Zone Alignment: A nice feature that would allow you to keep costs down, avoiding inter-zone tariffs and duplicated architecture, and enhance service resilience against zone failure. Part 2 of my article on AWS Availability Zone alignment covers the implementation process. Because nothing ever seems to be quite so straight forward when you mix DNS and AWS.
get your facts right
So, you may ask, how did we implement this? Probably because that’s the gist of the blog post… Well our configuration is governed by Puppet Hiera, and our frontends are Java based. On paper this sounds pretty trivial. We simply utilise the EC2 facts exposed in Facter, that Puppet can natively use to inject the Availability Zone into our Java parameters… Oh but wait, everything’s now in Hiera, which abstracts that and builds a hash from a YAML file that we plonk into a Puppet Template. Hmm, that’s a little more tricky, I guess we could write an exception in our template for this particular frontend? No, that’s messy and a path to ruin. The solution came a little more left field, with DNS search suffixes.
searching for a needle
So how can search suffixes help us? Well out of the box, an EC2 instance from Amazon (particularly with LTS Ubuntu), a default configuration for
/etc/resolv.conf is created, using the EC2 style
search eu-west-1.compute.internal. This means that any hostname you provide will do a lookup with that added on to the end as it continues up the hostname searching for a DNS response. One could simply create a search suffix entry with
eu-west-1a.domain generated by Puppet and lookups will always try and find something zone local first. Oh if only things could be that simple.
the final furlong
The last hurdle in this sprint came through dhclient’s behaviour in Ubuntu. There’s no direct concept of a ‘dhcp-client’ service that you can simply restart. Making changes to
/etc/resolv.conf does implement the change immediately, but only until the DHCP lease time has expired. See, there’s a daemon which gets initiated upon the start of
/etc/init/networking I might add!), and it sets
/etc/resolv.conf upon first startup, generated through
/etc/dhcp/dhclient.conf and the response from your DHCP server.. So any manual changes you make to
/etc/resolv.conf do not get persisted unless you edit
/etc/dhcp/dhclient.conf (or the DHCP server behaviour)! Until then you will get back the default configuration provided by the DHCP server. And, because we’re in AWS, all instances receive their IP from DHCP outside of user control.
oh if only
Guava. See, if you want to do clever things with DNS search suffixes, you have to be prepared for some pain. Because we’re effectively passing an incomplete hostname to our Java application as the caching layer hostname, Guava decided that it wasn’t a valid domain. Consider that our app was configured to use a hostname like
cache, and that gets resolved to
cache.eu-west-1a.domain, well Guava’s ‘HostSpecifier‘ only sees the host before it’s resolved, and it does some overzealous domain name validation. Hideously, and this is where the actual hack comes in, a quick fix was to compose a new fictitious hostname (does not exist in DNS) that passes validation, and can then be resolved to the correct zone-local hostname once it leaves the app server. I nearly cried.
ok maybe I cried a little
Because that still wasn’t the end of the debacle. When the request finally hits our caching layer, we have some validation in place to ensure that the correct hostname has been used to request cached content. Of course, the DNS hack we put in place fell foul to this. Because the Java application adds the header to send to the caching layer, and it only knows about the incomplete hostname that’s been put in its configuration; the search suffix resolution isn’t performed until a later stage. So a workaround had to be implemented for that too! We now have a filter that accepts this fictitious hostname in our caching layer configuration.
why, what why? why?
Yeah, that does sound like a spectacular faff just to achieve something like Availability Zone affinity. True, if I could take that time back, I would! But this pain we went through allowed us to reliably achieve full Availability Zone redundancy in a distributed architecture, and that’s something we’re rather proud of. And the hack of course was only a temporary measure ahead of code changes to remove the HostSpecifiers where necessary.