Creating Basic Custom Matchers in ScalaTest

If you are working in a large Scala codebase, it can sometimes be challenging to reduce the complexity of your tests. ScalaTest offers a vast array of tools to write clean, readable tests — including its Matcher class for creating our own custom Matcher.

While ScalaTest documentation dives into great detail (covering more advanced topics of custom Matchers), it lacks an excellent example of basic implementation. This article will look at the basics for setting up a custom Matcher using some simple example code.

The Code

Let’s look at some code we’d like to test:

class AssessmentEvaluator(correctAnswers: Assessment) {
  def evaluate(assessment: Assessment): AssessmentResult = {
    ...
  }
}

case class AssessmentResult(pass: Boolean, failures: Set[Failure])
case class Failure(questionName: String, correctAnswer: Answer, answerProvided: Answer)
case class Assessment(questionOne: Answer, questionTwo: Answer, questionThree: Answer)
object Answer extends Enumeration {
  type Answer = Value
  val A, B, C, D = Value
}

Here we have a class that is responsible for evaluating an Assessment of multiple-choice questions and providing an AssessmentResult.

Our tests might look like:

class AssessmentEvaluatorTest extends AnyFunSuite with Matchers {
  test("should fail last question") {
    val correctAnswers = Assessment(A, C, B)
    val providedAnswers = Assessment(A, C, A)
    val expectedFailures = Set(Failure("Question Three", correctAnswers.questionThree, providedAnswers.questionThree))

    val assessmentResult = new AssessmentEvaluator(correctAnswers).evaluate(providedAnswers)

    assessmentResult.pass shouldBe false
    assessmentResult.failures shouldBe expectedFailures
  }

  test("should pass") {
    val correctAnswers = Assessment(B, C, B)
    val providedAnswers = Assessment(B, C, B)

    val assessmentResult = new AssessmentEvaluator(correctAnswers).evaluate(providedAnswers)

    assessmentResult.pass shouldBe true
    assessmentResult.failures shouldBe Set.empty
  }
}

Now, in our simple example, these tests are probably sufficient, but we can use them to explore how we would implement a custom Matcher.

Passing Test Case

Let’s start with the “passing” case. We want to simplify our assertion down to assessmentResult should pass. Here’s the basic structure to get us there:

trait AssessmentResultMatcher extends Matchers {
  def pass = new PassingAssessmentResultMatcher

  class PassingAssessmentResultMatcher extends Matcher[AssessmentResult] {
    def apply(assessmentResult: AssessmentResult): MatchResult = {
      ...
    }
  }
}

We define in our trait the property pass, which returns us our PassingAssessmentResultMatcher. By extending Matcher[AssessmentResult], we get our apply method, the one parameter of which will be the value of the left side of should in our expression assessmentResult should pass and must be of type AssessmentResult.

Inside our apply method, we can define our assertion logic and return a MatchResult:

class PassingAssessmentResultMatcher extends Matcher[AssessmentResult] {
  def apply(assessmentResult: AssessmentResult): MatchResult = {
    val assessmentPasses = assessmentResult.pass && assessmentResult.failures.isEmpty
    MatchResult(assessmentPasses, "Assessment did not pass.", "Assessment passed when it should not have.")
  }
}

If we now go back to our test suite, we can mix in our trait so that it’s available for us to use and clean up our passing test case:

class AssessmentEvaluatorTest extends AnyFunSuite with AssessmentResultMatcher {
  ...
  test("should pass") {
    val correctAnswers = Assessment(B, C, B)
    val providedAnswers = Assessment(B, C, B)

    val assessmentResult = new AssessmentEvaluator(correctAnswers).evaluate(providedAnswers)

    assessmentResult should pass
  }
}

Failing Test Case

For the failing test case, we need to make a few changes. We want to be able to check not only that assessmentResult.pass is not true, but also that we have the correct expected set of Failure. To do this, we need to pass more information to our Matcher. We will define another property and corresponding class in our AssessmentResultMatcher:

trait AssessmentResultMatcher extends Matchers {
  def pass = new PassingAssessmentResultMatcher
  def failWith(expectedFailures: Set[Failure]) = new FailingAssessmentResultMatcher(expectedFailures)

  class PassingAssessmentResultMatcher extends Matcher[AssessmentResult] {
    def apply(assessmentResult: AssessmentResult): MatchResult = {
      val assessmentPasses = assessmentResult.pass && assessmentResult.failures.isEmpty
      MatchResult(assessmentPasses, "Assessment did not pass.", "Assessment passed when it should not have.")
    }
  }

  class FailingAssessmentResultMatcher(expectedFailures: Set[Failure]) extends Matcher[AssessmentResult] {
    def apply(assessmentResult: AssessmentResult): MatchResult = {
      ...
    }
  }
}

Notice that this time we pass along a parameter to failWith, which is our set of expected Failure. That is then used in the construction of our FailingAssessmentResultMatcher. Now that we have this additional information, we can implement our assertion logic:

class FailingAssessmentResultMatcher(expectedFailures: Set[Failure]) extends Matcher[AssessmentResult] {
  def apply(assessmentResult: AssessmentResult): MatchResult = {
    assessmentResult.pass match {
      case true => MatchResult(false, "Assessment passed when it was expected to fail.", "")
      case false => {
        if(assessmentResult.failures == expectedFailures) {
          MatchResult(true, "", "") //Notice we provide no failure messages here becuase this is our passing branch of logic
        } else {
          MatchResult(false, s"Assessment failed but failures do not match. Expected: ${expectedFailures}; Actual: ${expectedFailures}", "")
        }
      }
    }
  }
}

We can now simplify our test for the failing case:

test("should fail last question") {
  val correctAnswers = Assessment(A, C, B)
  val providedAnswers = Assessment(A, C, A)
  val expectedFailures = Set(Failure("Question Three", correctAnswers.questionThree, providedAnswers.questionThree))

  val assessmentResult = new AssessmentEvaluator(correctAnswers).evaluate(providedAnswers)

  assessmentResult should failWith(expectedFailures)
}

Bringing it All Together

And here’s our final product:

class AssessmentEvaluatorTest extends AnyFunSuite with AssessmentResultMatcher {
  test("should fail last question") {
    val correctAnswers = Assessment(A, C, B)
    val providedAnswers = Assessment(A, C, A)
    val expectedFailures = Set(Failure("Question Three", correctAnswers.questionThree, providedAnswers.questionThree))

    val assessmentResult = new AssessmentEvaluator(correctAnswers).evaluate(providedAnswers)

    assessmentResult should failWith(expectedFailures)
  }

  test("should pass") {
    val correctAnswers = Assessment(B, C, B)
    val providedAnswers = Assessment(B, C, B)

    val assessmentResult = new AssessmentEvaluator(correctAnswers).evaluate(providedAnswers)

    assessmentResult should pass
  }
}

trait AssessmentResultMatcher extends Matchers {
  def pass = new PassingAssessmentResultMatcher
  def failWith(expectedFailures: Set[Failure]) = new FailingAssessmentResultMatcher(expectedFailures)

  class PassingAssessmentResultMatcher extends Matcher[AssessmentResult] {
    def apply(assessmentResult: AssessmentResult): MatchResult = {
      val assessmentPasses = assessmentResult.pass && assessmentResult.failures.isEmpty
      MatchResult(assessmentPasses, "Assessment did not pass.", "Assessment passed when it should not have.")
    }
  }

  class FailingAssessmentResultMatcher(expectedFailures: Set[Failure]) extends Matcher[AssessmentResult] {
    def apply(assessmentResult: AssessmentResult): MatchResult = {
      assessmentResult.pass match {
        case true => MatchResult(false, "Assessment passed when it was expected to fail.", "")
        case false => {
          if(assessmentResult.failures == expectedFailures) {
            MatchResult(true, "", "")
          } else {
            MatchResult(false, s"Assessment failed but failures do not match. Expected: ${expectedFailures}; Actual: ${assessmentResult.failures}", "")
          }
        }
      }
    }
  }
}

This is a simple example, but it shows the potential power that custom Matchers can have in your codebase.