Tim Spurling

safety among system files

i love version control

As you might have noticed if you’ve read any of my other posts, I’m a massive fan of version control. To the extent that any sort of work where it can’t be used gives me an almost-literal headache.

This post describes one recent situation where I’ve managed to make use of Git in an unexpected way. While it’s nothing unusual technically, it’s still quite a nice example in my opinion, as the work in question has become much safer and easier compared to how it might have gone if I hadn’t thought of doing this.

my last week

Recently I’ve been taking a brief holiday from the safe, organized (but slightly boring) city of Java-based software engineering, in the exciting (but stressful and deadly) wilderness of DevOps. As part of this, I have been modifying our Jenkins build config.

Despite being a tool mostly developed for developers, Jenkins is typically configured through forms in its web UI. With the configuration you make in these screens, it writes config.xml files for each task in its jobs folder. Conversely, it can be manually persuaded (through its main config screen) to reload these files, allowing you to apply changes made through direct editing. Not bad!

Obviously though, editing these files in place is quite awkward and potentially dangerous. The jobs folder contains quite a lot of other data for each task, and in any sensible deployment you’d have to have unusual privileges to actively fiddle with Jenkins’ data at all. Finally, editing the files incorrectly could presumably result in totally broken behaviour. Using any sort of version control in this situation would be awkward at best, and horrifically destructive at worst.

I had to find a way to take control of the files, and make the editing process safer.

step one: protecting jenkins

The most anxiety-inducing part of the situation for me was trampling around the data directory of our live CI server with a fully-privileged shell. I am by nature slightly clumsy, and also a pretty incompetent shell user by any old-school sysadmin’s standards (I grew up with Ruby so never saw the point in learning bash syntax properly).

The solution? Write one incredibly simple script to copy the files into my home directory on the server:

#!/usr/bin/zsh
for i in $(ls /var/lib/jenkins/jobs); do
        cp /var/lib/jenkins/jobs/$i/config.xml ~tim/jobs-config/$i
done

Now, the only direct interaction with Jenkins is to run that under sudo. No chance left to screw that up. Any editing of the files happens in my own shell, where I can’t break anything. And, to apply changes, I just use an opposite script that copies the files back.

step two: protecting my sanity

Of course, with this new folder that only contains the relevant files, it’s really easy to apply Git.

git init
git add -A
git commit -m “Original config before I started meddling”

There! Logic restored. And, as a bonus, other people’s changes made through the UI can be easily merged with stuff done through an editor and committed here—just run the copying script again and Git will show the UI-driven changes against the last commit, allowing them to be committed/branched/merged in whatever way is most convenient at the time.

step three: enhancing editing

My second favourite part of this process was realizing how easy this makes remote editing. With this simple repo in existence, all I have to do to edit the files on my desktop is just clone them!

git clone ssh://jenkins-server/~/jobs-config

Then can edit them in any way, push the changes back, and keep working on the server, with complete confidence.

My absolute favourite part was using git rebase -i and the overwriting script to revert the config back to a previous state when external problems made it necessary.

I love version control.

blog comments powered by Disqus