Testing C#.Net Async Code

Writing C#.Net async code can be a bit of a challenge, and writing tests around the code can frequently cause much pain and agony. In fact, conventional wisdom around the Internet seems to be that writing async tests is hard enough that it’s okay to avoid it.

I disagree.

Writing tests has enough benefits (most notably that it improves the quality of the code under test, and it allows us to refactor and add future features with less risk) that it’s worth taking the time to experiment with testing strategies to find something that works for your project.

Test Failures

The important thing to remember when testing async code is that you need to wait for background tasks to complete, and handle any exceptions that are thrown in tasks. Otherwise, you’ll get mysterious test failures. Some examples:

• You don’t wait for tasks to complete, so assertions fail because you are asserting before the work has completed.
• You don’t wait for tasks to complete, then the test runner attempts to clean up after it’s done running all the tests, resulting in an exception like this:

  System.AppDomainUnloadedException: Attempted to access an 
  unloaded AppDomain. 
  This can happen if the test(s) started a thread but did not stop it. 
  Make sure that all the threads started by the test(s) are stopped 
  before completion.

• In bigger test suites with lots of tests, instead of the AppDomainUnloadedException, you might get an ObjectDisposedException, or a NullReferenceException before the test suite completes, as background threads attempt to access memory in classes that have been disposed or reinitialized for a later test. You’ll see these exceptions in the test runner’s output window.

Test Solutions

Let’s say we want to write some tests for the following code:

        public RelayCommand LoadCommand 
        { 
            get { return new RelayCommand(load); }
        }

        private readonly string[] listOfAddresses = new[] { "Happy", "New", "Year"};
        private async void load()
        {
            // Load each string asynchronously
            var tasks = listOfAddresses.Select(loadOneString).ToList();
            
            // Wait for all tasks to complete before adding a final message
            var allStrings = await Task.WhenAll(tasks);
            var completeMessage = String.Join(" ", allStrings);
            Messages.Add(completeMessage);
        }

        private readonly Random random = new Random();
        private async Task loadOneString(string url)
        {
            var taskFactory = new TaskFactory();
            var task = taskFactory.StartNew(() =>
                {
                    // Normally some meaningful work would be done here. 
                    // For our purposes we'll just wait a random amount of time.
                    var timeToWait = (int)((random.NextDouble() + 1)*2000);
                    Thread.Sleep(timeToWait);
                });

            // After the waiting is over we'll add to our list of messages
            await task;
            Messages.Add(url);

            return url;
        }

        private readonly ObservableCollection messages = new ObservableCollection(); 
        public ObservableCollection Messages
        {
            get { return messages; }
        }

This code is taken from a view model for a WPF application. The RelayCommand is databound to a button in my view. When the user clicks the button, the load() method loads a collection of strings — in the real world, we might be doing something like hitting various web servers for data. I want to to ensure that when I execute the RelayCommand, I ultimately end up with the correct results in the Messages property.

Here’s a test that definitely does not pass:

        [TestMethod]
        public void BadTest()
        {
            var underTest = new MainWindowViewModel();

            underTest.LoadCommand.Execute(null);

            // This fails because we haven't waited for underlying tasks to complete before asserting
            Assert.AreEqual(4, underTest.Messages.Count);
        }

How do we get the test to pass? Here are two different methods.

1. Store a Reference to the Task

One technique that has worked for me in situations like this is to store a reference to running tasks in a property that can be accessed in a test. I can modify my view model code like this:

        private void load()
        {
            RunningTask = loadAllStrings();
        }

        private async Task loadAllStrings()
        {
            // Load each string asynchronously
            var tasks = listOfAddresses.Select(loadOneString).ToList();
            
            // Wait for all tasks to complete before adding a final message
            var allStrings = await Task.WhenAll(tasks);
            var completeMessage = String.Join(" ", allStrings);
            Messages.Add(completeMessage);
        }

        // For testing purposes
        public Task RunningTask { get; set; }

And now I can write my test like this:

        [TestMethod]
        public void TestThatWaitsForTaskToComplete()
        {
            var underTest = new MainWindowViewModel();

            underTest.LoadCommand.Execute(null);

            // Wait for task to complete
            underTest.RunningTask.Wait();

            Assert.AreEqual(4, underTest.Messages.Count);
        }

It’s important to store the correct task in the RunningTask property. Recall that the code after an await keyword can itself potentially be run on a background thread (depending on what the underlying TaskScheduler decides), so we want to wait for the await code to complete before we make any assertions. If I did something like this:

        private async void load()
        {
            // Load each string asynchronously
            var tasks = listOfAddresses.Select(loadOneString).ToList();
            
            // Wait for all tasks to complete before adding a final message
            RunningTask = Task.WhenAll(tasks);
            var allStrings = await RunningTask;
            var completeMessage = String.Join(" ", allStrings);
            Messages.Add(completeMessage);
        }

My test still fails because RunningTask.Wait() returns after the tasks from loadOneString() complete, but before the await code in load() has executed.

2. Create a Special Task Scheduler

Another approach that I’ve used to test async code is to force it to run synchronously. This is done by writing a custom TaskScheduler that forces all code to run on the current thread, then supplying that TaskScheduler to my code when testing it. My custom TaskScheduler looks like this:

    public class TestTaskScheduler : TaskScheduler
    {
        protected override void QueueTask(Task task)
        {
            TryExecuteTask(task);
        }

        protected override bool TryExecuteTaskInline(Task task, bool taskWasPreviouslyQueued)
        {
            return TryExecuteTask(task);
        }

        protected override IEnumerable GetScheduledTasks()
        {
            return Enumerable.Empty();
        }

        public override int MaximumConcurrencyLevel { get { return 1; } }
    }

I now have to modify my code somewhat like this:

        private async Task loadOneString(string url)
        {
            var taskFactory = new TaskFactory(TaskScheduler);
            var task = taskFactory.StartNew(() =>
                {
                    // Normally some meaningful work would be done here. 
                    // For our purposes we'll just wait a random amount of time.
                    var timeToWait = (int)((random.NextDouble() + 1) * 2000);
                    Thread.Sleep(timeToWait);
                });

            // After the waiting is over we'll add to our list of messages
            await task;
            Messages.Add(url);

            return url;
        }

        // For testing purposes
        private TaskScheduler taskScheduler;
        public TaskScheduler TaskScheduler
        {
            get { return taskScheduler ?? TaskScheduler.Default; }
            set { taskScheduler = value; }
        }

When I create the TaskFactory in loadOneString(), I tell it to use the TaskScheduler indicated in my class’s TaskScheduler property. If I haven’t set one, the default TaskScheduler provided by .Net will be used. Now I can write my test like this:

        [TestMethod]
        public void TestThatUsesCustomTaskScheduler()
        {
            var underTest = new MainWindowViewModel
                {
                    TaskScheduler = new TestTaskScheduler()
                };

            underTest.LoadCommand.Execute(null);

            Assert.AreEqual(4, underTest.Messages.Count);            
        }

Conclusions

Both async testing methods presented here have their pros and cons. I like the Task reference method because it forces me to make explicit in my tests where things are happening on a background thread. I have to fully understand my code in order to get my tests to pass. It also helps me to carefully consider when I should put code on a background thread.

On the other hand, the TaskScheduler method forces all code to run synchronously in my tests, and when exceptions are thrown I can easily see where and diagnose the problem. However, in the TaskScheduler method, the code is running differently than it does in production, which should be avoided in tests if possible.

What tools and tricks are you using to test your C#.Net async code?