Garry Wilson

cutting the strings – getting rid of puppetmasters: part 1

To manage the many EC2 instances we run here at MetaBroadcast, and all of the software and code that runs on those instances, we’ve been using Puppet. It’s fair to say, Puppet doesn’t get a lot of love around these parts. It’s generally treated with fear, hatred and/or mistrust.

Our Puppet usage involves having two centralised puppetmaster instances, which all of our instances check-in with every 30 minutes. We bake custom AMIs (essentially, a startup copy of an operating system) with the necessary settings in place to have instances make their first check-in.

We use user-data in our instance launch configuration to tell Puppet what environment an instance is (prod or stage), and what role it should request from the puppetmasters on its next check-in (“Hey! I’m a Mongo database!”).

cutting the strings

While this all usually works, it’s not very elegant; we don’t want to have to create new AMIs often so as to avoid stale startup configuration, we don’t want the fate of hundreds of instances depending on a few puppetmasters (especially given Puppet’s rare but deadly memory leaks), and no one really enjoys wrangling Puppet code to set things up properly.

Our impending Kubernetes migration gives us the perfect opportunity to Pinocchio our way out of the current setup. Kubernetes will do the heavy lifting of moving software and code around depending on what resources are available where, so all we really need to manage is a small set of tasks such as setting up users and their SSH keys, and getting from a fresh OS AMI to having Kubernetes up and running.

a new challenger approaches

Out of the many words that comes up when someone’s having a particularly hard time with Puppet, and one of the few I’d be happy to repeat here, is Ansible. Like Puppet, it can help us set up the instances we use, but it doesn’t depend on having centralised servers such as the Puppetmasters.

Ansible has the added benefit of not using a language of its own making. We configure instances in Ansible with yaml, which coincidentally is what we’ll be using to define services in Kubernetes. Synergy!

We can use a great feature called Ansible-Pull. This allows Ansible to get its configuration from a git repo, and apply to the local instance. We’d replace the puppetmasters with the git repository, and give the Ansible instances a deploy key so they have read-only access.

So — no puppetmasters, replacing Puppet markup with yaml, and one last improvement while we’re at it: getting rid of the need to bake AMIs.

cloud-init: it’s super effective!

One feature that we’ve never really used much before is cloud-init, a service most modern cloud/EC2-friendly operating systems include that can manage some configuration at startup. As mentioned, we pass things like a server’s role and environment in via user-data, but we can also pass in an entire script that can be run on startup to configure the system.

This basic cloud-init script only has a small list of things to do — install git and Ansible, create the deploy key for the Ansible repo, and (for bonus points) set the EC2 credentials so that Ansible can find out a bit about the instance it’s running on.

This setup seems much nicer on paper already. My next blog post will cover the how; I’ll show the cloud-init script I’ve created, and a simple Ansible playbook to start with.

If you enjoyed the read, drop us a comment below or share the article, follow us on Twitter or subscribe to our #MetaBeers newsletter. Before you go, grab a PDF of the article, and let us know if it’s time we worked together.

blog comments powered by Disqus