What’s Different About Property-Based Testing
Property-based testing isn’t a replacement for any of the other types of testing you might know. It just lets us test things in a way that’s impractical with other styles of testing.
With a traditional unit test, you supply a fixed set of inputs and assert that a function call returns a fixed value for those inputs. Each time you run the test suite, it uses the same inputs and the same outputs.
With property-based testing, you first define a property. Then the testing library will generate many random inputs and verify that your property holds for all of them. If you aren’t used to thinking about your functions in terms of properties, then it might not be clear what this actually means.
Identifying a Property to Test
If you think back to when you took math, functions were often defined in terms of properties. A function that describes addition over a field is required to be associative and commutative. A function that’s invertible must be surjective (onto) and injective (one-to-one). These are all good candidates for a property-based test.
Another simple property that you might want to test is idempotency. That means for a function f, f(f(x)) = f(x) for all values of x. One example of an idempotent function is the absolute value function.
What’s Wrong with a Bunch of Examples?
Even after going through all that trouble, you aren’t getting as much as you think you might be out of an exhaustive list of examples. Why? Because someone can always replace the function under test with a switch statement that does no computation, but merely returns the expected value for each set of expected inputs without breaking the test.
That’s not something that’s likely to happen in real life, but here’s something that might. Say you solve a problem with a naive and slow but correct algorithm in a low-traffic area of your application where performance isn’t critical. Over time, the low-traffic area gains traffic until your algorithm becomes a bottlenenck.
Along comes a well-intentioned developer with a knack for optimization who is tasked with fixing your slow algorithm. After a little bit of research, the developer comes across a blazingly fast approach that will only work on a restricted domain. The function is then changed to check for and handle special cases at the top, restricting the domain. Once those have been ruled out, the general code runs.
This is a very common pattern with highly optimized code. But if the well-intentioned developer forgets to handle one of the special cases, then your function will no longer be correct. Unless the developer adds a spec for that case, the tests won’t catch it, either.
Property-Based Testing to the Rescue
With a property-based test, you are able to express the idea that a function should satisfy some property. The testing library will automatically handle generating random examples and running them through your function.
Since it automatically finds the edge cases, it will make your tests less verbose and easier to write. When a test does fail, the library will go a little bit further to try to find the minimal failing example. This makes it easier to track down the problem without getting bogged down with a bunch of potentially unimportant details.
Have you tried property-based testing? I’d like to hear about your experiences.