Retrospective: “Building a Virtual Appliance – Repeatably”

Article summary

A few years back, I wrote a post describing how we used Chef and other tools to build a virtual machine “appliance.”

The short version is that we tried to make a copy of everything we were installing so that if we needed to make a point release later, we’d be able to reproduce the build. We made sure that we mirrored all external dependencies, downloaded all the RPMs, and kept the DVD ISOs around.

It’s been a few years now, and I had cause to go back to it. Spoiler: It didn’t really work.

What Went Wrong

“If you wish to make an apple pie from scratch, you must first invent the universe.” – Carl Sagan

It was a really solid attempt, but the approach is fundamentally flawed. It assumes that you can pack a box with everything you need, put it on a shelf, and it’ll keep forever. That doesn’t work, though, because you can’t totally isolate things. You always need something that can’t be put in a box.

In this instance, the important things outside the box were Chef, VirtualBox, Vagrant, and all of their dependencies.

Boom.

In the intervening years, a new OSX version (and compiler) broke the old archived version of the JSON RubyGem’s native component–which that version of Chef and Ruby depend on.

Boom.

So we upgraded those pieces. But then it cascaded further as I spent time ripping through our old configuration and updating things to work with the new Chef and Ruby. And VBoxManage. And a plethora of other tools.

Boom.

So I gave up.

When we originally designed all of this, we made the assumption that by freezing the dependencies that we could, the possible changes would be isolated in lower-risk, more slowly moving components. Well, I played those odds and lost. It took far too long to bring things back up to speed.

The Lesson

I built a complicated system (an archiving / mirroring strategy) to solve a complicated problem (dependency management). However, everyone I work with understands dependency management, where the pitfalls are, and how the standard tools work. No one I work with understands the complicated system I built.

That caused friction and made the project more difficult to work on and update. The gains never really materialized because we didn’t need to take advantage of the system to create a point release.

I’ve since ripped it all out and greatly simplified our build process. I’ve decided to tackle the dependencies problem as a cultural problem rather than a technical one. We simply add a task to our backlog periodically to make sure that we review our dependencies and things don’t fall too far behind.

In general, I recommend keeping a problem your whole team understands instead of trying to solve it with a solution no one understands.