Migration Testing in Rails

Migrations are one of the most useful additions Rails provides. Knowing that you can rely on them working is good too. Recently we’ve released a plugin, migration_test_helper, which makes it easier to test the migrations you create for a Rails app individually and as a whole.

1 2 

 ./script/plugin install   svn://rubyforge.org/var/svn/migrationtest/tags/migration_test_helper

Full Migration Test

Let’s start by creating a migration test that proves we can migrate from an empty schema to the latest version.

1 2 3 4 

 ./script/generate full_migration_test        create  test/migration       create  test/migration/full_migration_test.rb

This will create test/migration/full_migration_test.rb. migration_test_helper stores migration tests in their own directory (next to unit, functional and integration) and provides a rake task to run them:

1 2 3 4 5 6 7 8 

 rake test:migrations   ...   implement me.   1 tests, 1 assertions, 1 failures, 0 errors  rake aborted!

After filling in a few helper methods, this test will:

1. drop all tables in the test database
2. run your migrations from version 0 to the latest version
3. assert the entire schema is as expected
4. assert any default data was loaded

The migrate helper has the same behavior as running rake db:migrate against your test database.

Let’s update our new migration test to verify our schema is built (say our app has a users, groups and groups_users table). There are two methods stubbed out in the full_migration_test for us to implement: see_full_schema and see_default_data.

see_full_schema is where we should put our assertions about what the schema looks like:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 

 def see_full_schema    assert_schema do |s|      s.table :users do |t|        t.column :id,    :integer        t.column :login, :string        t.column :email, :string      end       s.table :groups do |t|        t.column :id,    :integer        t.column :name,  :string      end       s.table :groups_users do |t|        t.column :group_id, :integer        t.column :user_id,  :integer      end    end  end

The assert_schema helper will fail if any table isn’t specified or found, or if any table’s column isn’t specified or found (with the matching type). This can be handy for catching mistakes and typos when creating migrations which modify the schema. You can test-drive your schema changes by, first, editing an assert_schema call in a test like the one above, then, creating a migration to accomplish that change.

This style of migration test can also be useful for verifying any default data migrations are working properly. Say your application adds an ‘admin’ user in the ‘Super User’ group through a migration to ensure a new installation can be logged into by its owner. If we fill in the see_default_data method with the following, we can prove that this happens:

1 2 3 4 5 6 7 8 

 def see_default_data    the_admin = User.find_by_login('admin')    assert_not_nil the_admin, "default admin user not added"    super_group = Group.find_by_name('Super User')    assert_not_nil super_group, "default Super User group not added"    assert the_admin.groups.include?(super_group),     "admin user doesn't belong to the Super User group"  end

Individual Migration Tests

Migrations which modify data that isn’t loaded by another migration can be more difficult to test. Say our users table has some entries with bad email addresses in it, and we want to make a migration to detect and blank out those invalid email addresses. First, let’s create the new migration, but not implement it:

1 2 3 4 

 ./script/generate migration remove_junk_email_addresses       exists  db/migrate      create  db/migrate/042_remove_junk_email_addresses.rb

Now let’s make an individual test for this migration. If given a migration number, instead of a name, the migration_test generator will create a test targeted at a particular migration.

1 2 3 4 

 ./script/generate migration_test 42       exists  test/migration      create  test/migration/042_remove_junk_email_addresses_test.rb

The generated test method will look something like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 

 def test_migration_should_remove_junk_email_addresses    drop_all_tables     migrate :version => 41     flunk "TODO: setup test data"      migrate :version => 42     flunk "TODO: examine results"      migrate :version => 41     flunk "TODO: examine results"  end

Since there aren’t default users added by previous migrations that have examples of ‘good’ and ‘bad’ email addresses to test against, we’ll have to add some for testing right before our migration runs. Then we can see that they’re updated once we’ve run our new migration.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 

 def test_migration_should_remove_junk_email_addresses    drop_all_tables     migrate :version => 41     User.reset_column_information     good_user = User.create!(:email => '[email protected]')    bad_user = User.create!(:email => 'what? email?')     migrate :version => 42     good_user.reload    assert_equal '[email protected]', good_user.email,     "valid email should have been left alone"    bad_user.reload    assert_equal '', bad_user.email,     "bad user should have no email set"      migrate :version => 41     flunk "TODO: examine results"  end

This test will migrate us all the way up to the migration right before ours. Here we stop to add some data which we can use to exercise what our migration 042 is supposed to do. Then we finish by migrating up to our version and see it had the desired effect on the users email addresses. The User.reset_column_information call ensures that when we talk about the User model in our test it doesn’t have any stale idea of what its columns are.

The only thing left is that migrate to version 41. What’s this for? You should test your down migrations, even if it does nothing. The down part of a migration is something you rarely need, but when you need it, you REALLY need it to behave correctly, eg. during a rollback.

In our case, let’s say we don’t care about restoring the old broken email addresses, and not do anything. We’re not proving much, but, we are proving that we didn’t write something stupid in there that’ll stop a further (more critical) down migration during a rollback:

1 2 3 4 5 6 7 8 9 10 11 12 13 

 def test_migration_should_remove_junk_email_addresses     ...     migrate :version => 41     good_user.reload    assert_equal '[email protected]', good_user.email,     "valid email should have been left alone"    bad_user.reload    assert_equal '', bad_user.email,     "bad user should have no email set"  end

Why Bother?

Testing your migrations can save you time and help you sleep. If you catch a problem with a migration while test-driving its creation, your development database won’t have to suffer being dropped down to version 0 then back up again and again while you try to figure out what’s wrong.

It’s better to know sooner than later when your migrations stop working. Maybe you’ll realize that it doesn’t matter since all production machines and developers have already migrated past that point. If that’s not the case, however, you have a problem that needs to be fixed before you deploy.

Some of the difficulties with migrations that can become apparent when trying to responsibly test them can be lessened by trying to follow some guidelines when possible:

Try to avoid using your model classes in migrations

Even though it is often much quicker to use your models and their associations to implement a data migration, it can be more painful down the road if that model’s table, columns or associations change or get removed. If you can stick with gross SQL using helpers on ActiveRecord::Base.connection like select_all, insert, select_value, etc. then your migration will be more likely to not break later on as your application evolves.

Try to get rid of old migrations

If you get to a point where every production machine and development workstation is beyond, say, schema version 33, try to collapse migrations 1-33 into one migration. This can be done by running your migrations from 0 => 33, and making your 033 migration create the schema and add any default data that exists at that point. To save time you can try taking a SQL dump of the entire database at that point, and simply have your “genesis” 033 migration load that.

This trick is much easier to accomplish if you have tests in place to ensure that your ‘compression’ of un-needed migrations didn’t miss anything.

Conclusion

Glenn Vanderberg gave a talk at RailsConf 2006 (which I missed) that seemed to discuss some of these problems. In it, he emphasized the importance of data migrations and down migrations, which I agree is worth pointing out. When you don’t need migrations anymore, you should find a way to get rid of them, and when you DO need them to work, you should make sure they do.