Unit Tests for Everything Else: nginx, Apache, & Virtual Hosts

I’ve always been a fan of unit testing and TDD. And with the way frameworks have matured over the years, that’s only gotten easier to do effectively. Despite that, one area is still very often without tests: servers, configuration, and deploys. Even the simplest applications tend to have an alphabet soup of technologies running to get the code onto the server, to handle SSL and virtual hosts, to rewrite URLs, and do all sorts of other tweaks before the users see anything.

All too often, this ancillary code goes untested. Sometimes, this is a problem.

Recently the SME Toolkit (a web site we perform operational support and maintenance on for the IFC ) moved onto a new hosting platform. The Toolkit is a Ruby on Rails application that sits behind nginx. Here’s a few facts about our configuration:

  • The nginx configuration totals over 15,000 lines of ‘code’. Granted, most of this is generated and repetitive, but it’s still complicated.
  • Nginx responds to 455 domain names.
  • We have over 2,500 lines of code for Capistrano driving templates and configuration for a plethora of critical services.

Needless to say, there’s enough complexity to warrant tests. However, this is not a trivial task: normally when unit testing application code there’s an isolated unit for which we can provide inputs and expect known outputs. For most deployments, things are not so simple.

We’ve been tackling this problem a piece at a time with various solutions for tests and monitoring. Today I’d like to show you one of the small scripts we have for validating some our rewrite and redirect rules:

require 'rubygems'
require 'http-access2' # from gem httpclient

tests = {
  # A simple test: GET a URL and expect a successful response
  "Home page" => {
    :get => "http://www.myapp.com/",
    :result => 200

  # A simple test: GET a URL and expect a specific response code
  "Home page" => {
    :get => "http://www.myapp.com/whatever",
    :result => 403 # forbidden

  # More complicated: GET a URL and expect a redirect to an HTTPS login page
  "Restricted area" => {
    :get => "http://www.myapp.com/private",
    :result => "https://secure.myapp.com/login"

require 'uri'
require 'pp'

test_count = 0
test_failures = 0

tests.each do |test_name, test_config|
  puts "Running: #{test_name}"
    uri = URI.parse(test_config[:get])
    client = HTTPClient.new
    result = client.get(uri)

    if String === test_config[:result] # assert redirected to a URL
      if !result.status_code.to_s[/\A3..\Z/]
        puts "  Fail: Did not redirect, was [#{result.status_code}]"
        test_failures += 1 
      elsif result.header['Location'].to_s != test_config[:result]
        puts "  Fail: Redirected to:\n got:     [#{result.header['Location']}]\nexpected: [#{test_config[:result]}]\n"
        test_failures += 1 
    elsif Numeric === test_config[:result]
      if result.status_code != test_config[:result]
        puts "  Fail: Wrong response received [#{result.status_code}], expected [#{test_config[:result]}]"
      puts "don't know how to validate #{test_config[:result].pretty_inspect}"
      test_failures += 1
  rescue Exception => ex
    puts "Fail: #{ex.class.name}: #{ex.message}"
    puts ex.backtrace
    test_failures += 1

  test_count += 1

puts "#{test_count - test_failures} / #{test_count} tests passed"
exit (test_failures == 0 ? 0 : 2)

One deficiency in this particular script is that it does not include automation for testing logins or any such complex logic. Despite that restriction, for us it’s been valuable because we have a large number of redirects and rewrites that interact in sometimes surpising ways and are hard to debug.

This script (and scripts like it) can be written fairly quickly and easily tied into long-term monitoring solutions like Nagios, or even just cron jobs. Having this validation apply for the life of the project is helping to prevent an entire class of bugs.

What sort of simple scripts and monitoring have you found valuable?

  • Travis Tilley says:

    I had just recently complained that nobody really seems to be doing this, at least not comprehensively. Even the solution above doesn’t give you any insight into whether or not a rewrite is doing what you expect it to, just that the server responded with a status code that suggests correctness.

    The way I went about testing and refactoring a complex set of rewrite rules recently was to create new vhosts that replaced our normal wsgi app with a testing wsgi that returned various chunks of the environment as JSON, and then testing in the client if the response met our expectations. You could easily replace wsgi and mod_wsgi in the above with rack and passenger to the same effect.

    This obviously is only useful when the request goes through to the application and not when you’re hitting static content with a ‘last’ rule, etc. :-/

    Another option (for apache, anyways) is to enable the rewrite log, pump up the logging level, and have the client read from it as requests are being made. That way nothing about the process is getting missed, as it doesn’t depend on a request falling through to a script that serves back that data. It sounds like it would be painful to implement though.

    It’s good to know we’re not the only ones with complex enough rewrite rules to require some method of unit testing them.

  • Mike Swieton Mike Swieton says:

    Hi Travis,

    Thanks for your comments. You’re right that this doesn’t go very deep. My first preference would definitely be to setup some cucumber and celerity tests, but the level of effort required for that is much higher.

    Most of our apps are written to handle rewrites inside of Rails (or the appropriate framework). That just wasn’t an option here for various reasons.

    I think a few simple things could be added, such as looking for various strings on the page that could potentially make this much more valuable. It’s a start at least.

  • Comments are closed.