Knocking Out Bugs with greatest

I looked into various options for unit-testing C, but wasn’t satisfied with any of them. First, I looked at Unity. While its suite of tools work quite well, they depend on Ruby/rake/YAML, require creating a full project configuration, and take on many other responsibilities. I wanted a tool for testing embedded C, not a whole build and testing framework, and having to define a whole project adds too much overhead for testing small programs.

Then, I looked at minunit, which is only a few lines of code. But while it doesn’t impose any particular workflow or toolchain, it stops after the first test failure, and doesn’t provide any way to control which tests run. I appreciate its transparency, but wanted something that provided more infrastructure.

After ruling out a couple others because they required dynamic allocation, I decided to write greatest. It doesn’t depend on anything beyond ANSI C or do any allocation, and it keeps boilerplate to a minimum. Since it starts quickly and can run only an individual suite or test, it allows rapid iteration during development. (With one Emacs keybinding, I can re-compile and re-test the code I’m working on.) It also tracks pass/fail/skip counts, suite & test runtimes, and files & line numbers for failing assertions.

greatest Setup

All of greatest is contained in one header, greatest.h. This should either be copied into the codebase or available via include paths.

First, there should be a .c file for the test runner, which will have its own main(). A minimal example looks like this:

 #include "greatest.h"
 
 /* Expand to all the definitions that need to be in
    the test runner's main file. */
 GREATEST_MAIN_DEFS();
 
 int main(int argc, char **argv) {
     GREATEST_MAIN_BEGIN();      /* command-line arguments, initialization. */
     RUN_SUITE(example_suite);   /* run a suite */
     GREATEST_MAIN_END();        /* display results */
  }

Since a suite is just a function defined with SUITE(name), suites can be put anywhere, as long as they are available at link-time. Just include an extern declaration, e.g.

 extern SUITE(example_suite);

for the test runner. A file with a suite definition needs:

 #include "greatest.h"
 
 TEST example() {
     PASS();
 }
 
 GREATEST_SUITE(example_suite) {
     RUN_TEST(example);
 }

Test Cases

Of course, most of the actual testing code is in the test cases. Every test is a function with zero or more assertions, then a call to PASS(), FAIL(), or SKIP(). (There are variants that return a custom message, PASSm(msg), FAILm(msg), and SKIPm(msg).)

There are currently the following assertions:

ASSERT(COND) / ASSERTm(MSG, COND)

Assert that (COND) evaluates to a true value.

ASSERT_FALSE(COND) / ASSERT_FALSEm(MSG, COND)

Assert that (COND) evaluates to a false value.

ASSERT_EQ(EXPECTED, ACTUAL) / ASSERT_EQm(MSG, EXPECTED, ACTUAL)

Assert that EXPECTED == ACTUAL. (If structures are being compared, use ASSERT with your own comparison function.)

ASSERT_STR_EQ(EXPECTED, ACTUAL) / ASSERT_STR_EQm(MSG, EXPECTED, ACTUAL)

Assert that strcmp(EXPECTED, ACTUAL) == 0.

Example Test Case

This test case (from heatshrink) checks that when compressing the string “abcdabcd”, the repetition is detected:

 TEST encoder_poll_should_detect_repeated_substring() {
     heatshrink_encoder *hse = heatshrink_encoder_alloc(8, 3);
     uint8_t input[] = {'a', 'b', 'c', 'd', 'a', 'b', 'c', 'd'};
     uint8_t output[1024];
     uint8_t expected[] = {0xb0, 0xd8, 0xac, 0x76, 0x40, 0x1b };
 
     size_t copied = 0;
     memset(output, 0, 1024);
     HSE_sink_res sres = heatshrink_encoder_sink(hse, input, sizeof(input), &copied);
     ASSERT_EQ(HSER_SINK_OK, sres);
     ASSERT_EQ(sizeof(input), copied);
 
     HSE_finish_res fres = heatshrink_encoder_finish(hse);
     ASSERT_EQ(HSER_FINISH_MORE, fres);
 
     ASSERT_EQ(HSER_POLL_EMPTY, heatshrink_encoder_poll(hse, output, 1024, &copied));
     fres = heatshrink_encoder_finish(hse);
     ASSERT_EQ(HSER_FINISH_DONE, fres);
 
     ASSERT_EQ(sizeof(expected), copied);
     for (int i=0; i<sizeof(expected); i++) ASSERT_EQ(expected[i], output[i]);
     heatshrink_encoder_free(hse);
     PASS();
 }

Suites

A suite is a function, defined with SUITE(name), that calls test cases with RUN_TEST(test_name). There are two hooks for executing code around tests in the current suite:

SET_SETUP(*CALLBACK, ENVIRONMENT)

SET_SETUP registers a function (of type void(CALLBACK)(void *environment)) and its environment (a closure) to be called before any subsequent test cases. Calling it with a NULL function pointer clears it.

SET_TEARDOWN(*CALLBACK, ENVIRONMENT)

Similarly, SET_TEARDOWN registers a function and environment to call after every test is run (whether they pass or not).

Example Suite

Here is an example suite, again from heatshrink:

 SUITE(encoding) {
     RUN_TEST(encoder_alloc_should_reject_invalid_arguments);
 
     RUN_TEST(encoder_sink_should_reject_nulls);
     RUN_TEST(encoder_sink_should_accept_input_when_it_will_fit);
     RUN_TEST(encoder_sink_should_accept_partial_input_when_some_will_fit);
 
     RUN_TEST(encoder_poll_should_reject_nulls);
     RUN_TEST(encoder_poll_should_indicate_when_no_input_is_provided);
 
     RUN_TEST(encoder_finish_should_reject_nulls);
 
     RUN_TEST(encoder_should_emit_data_without_repetitions_as_literal_sequence);
     RUN_TEST(encoder_should_emit_series_of_same_byte_as_literal_then_backref);
     RUN_TEST(encoder_poll_should_detect_repeated_substring);
     RUN_TEST(encoder_poll_should_detect_repeated_substring_and_preserve_trailing_literal);
 }

(Since a suite is generally just a list of tests, it can be generated by grepping test files for lines beginning with ^TEST and wrapping the names in RUN_TEST.)

Abbreviations

All of the symbols greatest exports are prefixed with GREATEST_ or greatest_, but if GREATEST_USE_ABBREVS is #defined to 1 (the default), most are aliased to non-prefixed names (e.g. ASSERT).

Parametric Testing

If compiled under C99 (-std=c99), greatest can run tests with arguments. This is useful for running tests with generated data, stress testing, randomized/fuzz testing, and so on. Instead of using RUN_TEST, use RUN_TESTp(test_name, [arguments...]). (This is only available in C99 because greatest needs its __VA_ARGS__ keyword to portably implement parametric testing.)

Mocking and Other Higher-Level Functionality

While greatest doesn’t provide any direct support for mocking, BDD, or other testing approaches, it also doesn’t exclude them. It’s designed to provide portable infrastructure for automated testing, in a sufficiently flexible way that it can be extended for project-specific testing styles. As testing practices continue to evolve, it will be able to adapt.