Writing a Fuzzing API with Clojure’s test.check

I’ve written before about testing CSS using a fuzzing API. Having relied on my fake, random API over the last several months, I’m confident that it’s a tool I’ll use on all my future projects. Besides supplying the app with random data, it’s also given me an unprecedented ability to customize the fake API’s behavior and test unusual and hard-to-reproduce scenarios. In this post, I’ll walk you through the key components of my API and some ways that I’ve used it.

I wrote my fuzzing API in Clojure. Clojure provides two essential capabilities: a REPL and the test.check library.

Create a Server

We’ll use Ring to create a simple HTTP server.

(ns fuzz-api.core
  (:require [ring.adapter.jetty :refer [run-jetty]]
            [ring.middleware.reload :refer [wrap-reload]]
            [ring.middleware.stacktrace :refer [wrap-stacktrace]]))

(defn item [req]
  {})

(defn items [req]
  {})

(defn handler [req]
  (let [f (condp re-matches (req :uri)
            #"^/api/items/\d+$" item
            #"^/api/items$" items
            (fn [_] {:status 404}))]
    (f req)))

(def app
  (-> handler
    wrap-reload
    wrap-stacktrace))

(defonce server
  (run-jetty #'app {:port 5003 :join? false}))

Starting a REPL with lein repl, the following will start the server:

user=> (require 'fuzz-api.core)

The server will now respond to requests on Port 5003, but it only gives empty responses. We’ll use Cheshire‘s cheshire.core/generate-string function to serialize Clojure data structures as JSON.

(defn item [req]
  {:body (generate-string {:id 5})})

(defn items [req]
  {:body (generate-string {:items [{:id 5}]})})

Generate Fake Data

To get random data, we’ll compose some test.check generators together. Require test.check’s generators under the alias gen inside the namespace’s :require block:

[clojure.test.check.generators :as gen]

Here’s a generator to create random IDs:

(def ids (gen/choose 0 1000000))

Generators are special test.check objects instead of normal values. To get a primitive value, use gen/generate:

(defn item [req]
  {:body (generate-string {:id (gen/generate ids)})})

Now every time we call /api/items/4, we’ll get an item with a random ID between 0 and 1,000,000.

To generate a list of items, we’ll want something that can generate whole items, not just IDs.

(def random-items
  (gen/fmap
    (fn [id]
      {:id id})
    ids))

gen/fmap takes a function and a generator. It passes values from the generator into the function and returns a new generator of the results. We can use random-items in our item function in place of creating a map:

(defn item [req]
  {:body (generate-string (gen/generate random-items))})

We can use random-items to create a list of items:

(defn items [req]
  {:body (generate-string
           (gen/generate (gen/vector random-items 20)))})

Our list of items isn’t fully random. It always returns 20 items. Let’s get lists of random lengths, too:

(defn random-list [generator]
  (gen/bind
    (gen/choose 0 50)
    (fn [n]
      (gen/vector generator n))))

Notice that we used gen/bind instead of gen/fmap. gen/fmap takes values from a generator and produces a new value (which then gets turned into a generator). gen/bind takes values from a generator and produces a new generator.

In this case, we’ll use gen/choose again to randomly pick a number between 0 and 50, then pass it to our function to create a generator of vectors with the chosen length.

Our random-list function is also an example of a higher-order generator, a generator that takes another generator as an argument.

Here’s how we use it:

(defn items [req]
  {:body (generate-string
           (gen/generate (random-list random-items)))})

That’s a very brief introduction to using test.check’s data generators. You can find more information in its docs.

Manipulate the Server

One of the chief pleasures of creating our server with Clojure is that we can change its behavior it at runtime. Because we started it from the REPL, we have access to the running instance and its code.

Let’s suppose that we want to see what happens in our app when it gets a response of 401 Unauthorized from the /api/items endpoint.

From the REPL, re-require our namespace to get all the changes we’ve made:

user=> (require 'fuzz-api.core :reload)

Replace the items functions with one that returns a 401:

user=> (alter-var-root #'fuzz-api.core/items
  #_=>   (fn [_] (fn [req] {:status 401})))

When you make an HTTP call to /api/items, it should now return a status of 401 Unauthorized.

To reset the server to its original state, reload the namespace again.

Experiment!

There’s a lot more that you can do with a random, REPL-driven API. You can delay server responses to see loading animations. You can make responses include certain text to see how search results are highlighted. You can freeze responses so they return exactly the same thing they did last. You can make them return in different amounts of time to test for race conditions. You can forward requests to another API. The tutorial above is just a taste.

The complete example code for this post is on GitHub.