Testing in Elixir with Multiple Environments

About a year ago, I was learning how to properly unit test Phoenix applications using the Mox library. The strategy I adopted for my side projects involves adding “module dependencies” to any module I want to unit test, and then using Mox to configure those dependencies during the tests.

While that approach is great for preserving good unit test hygiene, such as using explicit intra-module contracts, it isn’t always the simplest. One major hurdle I’ve encountered since writing that post is figuring out how to use both unit and integration tests in a sane way.

This may seem like an easy task, but here’s why it’s not: The module dependencies get configured during compilation, not during runtime.

@dependency Application.get_env(:my_app, :my_dependency)

@dependency is evaluated during compilation, so there’s no use in changing the environment during runtime.

Long story short, the best solution I’ve found is to define a new environment for each category of testing I need. Specifically, I define different Mix environments and then configure them with a corresponding file under the config/ directory.

Defining New Mix Environments

Our main goal here is to be able to run MIX_ENV=integration mix test, which should only run the integration tests, and with the correct environment configuration.

Run MIX_ENV=integration mix test in your Phoenix project as-is, and you’ll receive an error about a module that is “not loaded and could not be found.” That’s because the default Mixfile doesn’t load testing-related modules when MIX_ENV is set to anything other than :test.

Adding a new test environment

To fix that, start by opening up mix.ex. Notice that a couple of function clauses specifically match against :test, which won’t work for the new :integration environment. We’re going to loosen those constraints by using function guards. First, add an attribute somewhere near the top of the module:

@test_envs [:test, :integration]

Next, find the line that begins with defp elixirc_paths(:test)…, then change this clause to:

defp elixirc_paths(env) when env in @test_envs…

In your deps/0, find any dependencies that are only used in the :test environment, such as Mox. Change only: :test to only: @test_envs.

The config files

Now, if you run MIX_ENV=integration mix test, your tests still won’t run. You should see something like:

** (File.Error) could not read file "your-directory/my_app/config/integration.exs": no such file or directory

This is because config/config.exs loads an extra config file based on your Mix environment. We can fix this by simply adding a file under config/ named integration.exs:

use Mix.Config
import_config "test.exs"

# Here's where the two testing environments differ so that Mox is configured correctly:
config :my_app, service_module: MyApp.TheRealService

Some config files provided by Elixir and Phoenix use import_config at the bottom to do some dynamic configuration. Here, the import happens first so that we can override only enough configuration to run integration tests–namely, the module dependencies.

Separating the test environments

Now, you can run MIX_ENV=integration mix test and your tests should run. However, they probably won’t pass, because you just configured your module dependencies in a way that’s not compatible with your existing unit tests. That’s okay though; we don’t want to be able to run unit tests under the integration environment (or vice versa).

The next step is to only run the right tests depending on the Mix environment. In order to do that, let’s dig into the Mixfile again:

project/0 returns a keyword list, and one of those keys is :test_paths. Change that key’s value to test_paths(Mix.env), and add these functions:

defp test_paths(:integration), do: ["test/integration“]
defp test_paths(_), do: [“test/unit“]

Then, move your tests so that they’re under the right directory (test/unit or test/integration).

And viola! If you run mix test, you should get the exact same output as you did before making any changes. If you run MIX_ENV=integration mix test, it should re-compile your project for the new environment, but not run any tests until you add some in the new testing directory.

As a side note, this is not the only way to do things! I actually put my unit tests inside the lib/ directory, directly adjacent to the files of the modules they’re testing. For a module named example.ex, I put example.test.exs in the same directory. This requires changing the :test_pattern key inside of my Mixfile’s project definition to "*.test.exs", which isn’t a big deal.

A Clunky Approach

Setting MIX_ENV to run tests is a little awkward. Let’s add some custom mix tasks:

"test.all": ["test.unit", "test.integration"],
"test.unit": &run_unit_tests/1,
"test.integration": &run_integration_tests/1,

How do we implement those two functions? The following technique was borrowed from Ecto.

def run_integration_tests(args), do: test_with_env("integration", args)
def run_unit_tests(args), do: test_with_env("test", args)

def test_with_env(env, args) do
  args = if IO.ANSI.enabled?, do: ["--color"|args], else: ["--no-color"|args]
  IO.puts "==> Running tests with `MIX_ENV=#{env}`"
  {_, res} = System.cmd "mix", ["test"|args],
    into: IO.binstream(:stdio, :line),
    env: [{"MIX_ENV", to_string(env)}]

  if res > 0 do
    System.at_exit(fn _ -> exit({:shutdown, 1}) end)
  end
end

Now, if you want to implement an acceptance or non-functional test environment, you can reuse test_with_env/2.

Go ahead and run mix clean to delete cached build artifacts, then run mix test.all. You’ll see your project get compiled twice, once for each test environment. Neat!

What About Umbrella Projects?

I wrote a module named `mixfile_helpers.ex`, which defines anything that could reduce redundancy across Mixfiles. Each Mixfile begins with `Code.load_file(“mixfile_helpers.ex”)`; you can’t `import` or `alias` the module because Mixfiles are executed before your application is loaded.

Final Thoughts

What don’t I like about this approach? It seems like module dependencies should be configurable during runtime, but then it wouldn’t be possible to run tests asynchronously (because module implementation would change unpredictably). This might make CI run a bit slower, since it has to compile the app once for each testing environment.

If you have any experience with testing Elixir with multiple configurations, share your story in the comments.