This post presents a pleasantly hacky solution to a problem you might encounter with DNS, when migrating services between “classic” Amazon EC2 and a VPC.
We’ve recently been (gradually) moving our AWS-powered services from the shared EC2-Classic subnet into our own VPC subnet.
As Tom mentioned previously, each EC2 instance has two network interfaces—an external one and an internal one. This is important as it separates traffic for both billing (internal traffic is cheaper) and security (different firewall rules).
Normally this always magically works as we expect; the Amazon DNS servers resolve an instance’s default public hostname (e.g. ec2-176-34-218-135.eu-west-1.compute.amazonaws.com) to the internal address when queried from another AWS box, and to the external address otherwise.
Unfortunately, while the migration is in progress, the instances are no longer on the same subnet as each other; and internal DNS requests from one subnet about the other are sadly answered with the external IP.
For internal services requiring connections across the subnets, we have to temporarily switch to specifying hosts by their IP addresses instead of names, or to some other form of discovery—or, we must migrate all connected services at once.
As for us humans on the VPN, we can no longer even reliably SSH in to a named host either—when hosts are on the VPC and our VPN server is not, the hostname resolves to the external IP address, so the connection attempt happens over the external interface, where port 22 is closed.
The following script reads all of our DNS records from Route 53, and finds any chains of CNAMEs to an instance’s public hostname (such as ec2-176-34-218-135.eu-west-1.compute.amazonaws.com).
If called with the single argument “dump” and given appropriate rights (e.g. by sudo) it (reasonably safely) adds overrides to /etc/hosts, pointing all the hostnames in each chain directly to the internal IP address. Problem solved, for all services over the VPN!
So, maybe that will be helpful somehow. Twitter at us if you have any questions, comments, or information. Goodbye!