Unit Tests for Everything Else: nginx, Apache, & Virtual Hosts

I’ve always been a fan of unit testing and TDD. And with the way frameworks have matured over the years, that’s only gotten easier to do effectively. Despite that, one area is still very often without tests: servers, configuration, and deploys. Even the simplest applications tend to have an alphabet soup of technologies running to get the code onto the server, to handle SSL and virtual hosts, to rewrite URLs, and do all sorts of other tweaks before the users see anything.

All too often, this ancillary code goes untested. Sometimes, this is a problem.

Recently the SME Toolkit (a web site we perform operational support and maintenance on for the IFC ) moved onto a new hosting platform. The Toolkit is a Ruby on Rails application that sits behind nginx. Here’s a few facts about our configuration:

• The nginx configuration totals over 15,000 lines of ‘code’. Granted, most of this is generated and repetitive, but it’s still complicated.
• Nginx responds to 455 domain names.
• We have over 2,500 lines of code for Capistrano driving templates and configuration for a plethora of critical services.

Needless to say, there’s enough complexity to warrant tests. However, this is not a trivial task: normally when unit testing application code there’s an isolated unit for which we can provide inputs and expect known outputs. For most deployments, things are not so simple.

We’ve been tackling this problem a piece at a time with various solutions for tests and monitoring. Today I’d like to show you one of the small scripts we have for validating some our rewrite and redirect rules:

require 'rubygems'
require 'http-access2' # from gem httpclient

tests = {
  # A simple test: GET a URL and expect a successful response
  "Home page" => {
    :get => "http://www.myapp.com/",
    :result => 200
  },

  # A simple test: GET a URL and expect a specific response code
  "Home page" => {
    :get => "http://www.myapp.com/whatever",
    :result => 403 # forbidden
  },

  # More complicated: GET a URL and expect a redirect to an HTTPS login page
  "Restricted area" => {
    :get => "http://www.myapp.com/private",
    :result => "https://secure.myapp.com/login"
  },
}

require 'uri'
require 'pp'

test_count = 0
test_failures = 0

tests.each do |test_name, test_config|
  puts "Running: #{test_name}"
  begin
    uri = URI.parse(test_config[:get])
    client = HTTPClient.new
    result = client.get(uri)

    if String === test_config[:result] # assert redirected to a URL
      if !result.status_code.to_s[/\A3..\Z/]
        puts "  Fail: Did not redirect, was [#{result.status_code}]"
        test_failures += 1 
      elsif result.header['Location'].to_s != test_config[:result]
        puts "  Fail: Redirected to:\n got:     [#{result.header['Location']}]\nexpected: [#{test_config[:result]}]\n"
        test_failures += 1 
      end
    elsif Numeric === test_config[:result]
      if result.status_code != test_config[:result]
        puts "  Fail: Wrong response received [#{result.status_code}], expected [#{test_config[:result]}]"
      end
    else
      puts "don't know how to validate #{test_config[:result].pretty_inspect}"
      test_failures += 1
    end
  rescue Exception => ex
    puts "Fail: #{ex.class.name}: #{ex.message}"
    puts
    puts ex.backtrace
    puts
    test_failures += 1
  end

  test_count += 1
end

puts
puts "#{test_count - test_failures} / #{test_count} tests passed"
exit (test_failures == 0 ? 0 : 2)

One deficiency in this particular script is that it does not include automation for testing logins or any such complex logic. Despite that restriction, for us it’s been valuable because we have a large number of redirects and rewrites that interact in sometimes surpising ways and are hard to debug.

This script (and scripts like it) can be written fairly quickly and easily tied into long-term monitoring solutions like Nagios, or even just cron jobs. Having this validation apply for the life of the project is helping to prevent an entire class of bugs.

What sort of simple scripts and monitoring have you found valuable?