What state are your properties in?

My last blog ‘Prop’ up your tests with test.check walked through an example of using Property Based Testing (PBT) on a very simple RESTful API.

However, this approach had limitations.

I could test the properties of each call on the API in isolation but what if I wanted to generate tests and assert about properties that spanned calls to the API.

I want to be able to generate random get, post and delete methods for randomly generated resources and be able to assert on these randomly generated commands. For example, if my PBT’s generate a post followed by a get for the same resource I should find the resource but if the tests generate a post, delete, get sequence for the same resource I should get a 404 (not found) on the get.

This gives me a couple of problems.

1. How to generate random streams of commands with appropriate arguments.
2. How to assert on properties based on the expected state of the resources on the server.

Stateful.Check

Fortunately there is a Clojure library for that.

Stateful.check allows me to model the state of any system and to generate random streams of commands with their associated pre and post conditions.

Due to a change in the implementation of the rose tree in the latest version of test.check I needed to use ‘Michael Drogalis’ fork of stateful.check

Stateful.check uses specification maps to model state, define how to generate arguments, check pre and post conditions, etc.

Stateful.check specifications have two parts: an abstract model (state) and a real execution. It took me some trial and error to work out that the abstract model stores the functions you define to operate on  either the model or the real values as symbols and records the order of commands and only during the execution phase does it replay this stack of functions but this time actually evaluating them against the real values.

The model (state) is defined as a hash map and you supply the initial value of the model in the :init-state value of the overall specification.

The state map is used to model the state of the actual implementation. In my case the state map models the state of Customer resources on the server for my simple Customer RESTful API.

To use stateful.check I need to include it in my project.clj file:

(defproject
....
 :profiles
  {:dev {:dependencies [[javax.servlet/servlet-api "2.5"]
                        [ring/ring-mock "0.3.0"]
                        [cheshire "5.5.0"]
                        [org.clojure/test.check "0.9.0"]
                        [com.gfredericks/test.chuck "0.2.6"]
                        ; add stateful-check in dev profile
                        [mdrogalis/stateful-check "0.3.2"]]}})

Data wrangling with Clojure — Echo One

Clojure is a great language for wrangling data that is either awkwardly-sized or where data needs to be drawn from and stored in different locations. What does awkward-sized data mean? I am going to attribute the term “awkward-sized data” to Henry Garner and Bruce Durling. Awkward-sized data is neither big data nor small data and […]

via Data wrangling with Clojure — Echo One

‘Prop’ up your tests with test.check

I’ve been experimenting for a few months on and off with property based testing, otherwise known as generative testing. This blog is an attempt to show property based testing applied to the kind of business problems I deal with most days.

The principle of property based testing is to look for invariant ‘properties’ of a function under test, generate random test data for the function and verify that the ‘property’ of the function holds true for every generated test case. This contrasts with traditional testing that takes an ‘example’ based approach, i.e. explicitly coding each input and asserting on the expected output.

Property based testing (PBT) is a powerful technique that discovers edge cases more thoroughly than traditional ‘example’ based testing as randomly generated input tends to discover test cases that no human would think of. Also PBT can generate hundreds or even thousands of tests.

PBT is exemplified in John Hughes work in Haskell’s QuickCheck and the subsequent Erlang implementation of QuickCheck. QuickCheck has also been implemented in other languages notably, FsCheck for F#, ScalaCheck for Scala and, unsurprisingly, there’s an implementation for Clojure called test.check by Reid Draper. I am not going to go through a detailed description of the power of PBT in this blog but if you’re interested these talks by Reid (Reid Draper – Powerful Testing with test.check) and John Hughes (John Hughes – Testing the Hard Stuff and Staying Sane) are well worth checking out

However, I found PBT was not a substitute for example based tests but more a supplement. Example based tests can provide ‘developer readable’ documentation in a way that PBT doesn’t (or at least doesn’t for me). Also thinking of properties to test is hard. It takes quite a lot of thought and sometimes quite complex code in it’s own right to generate and verify randomly generated test data. I personally found it hard to come up with generic properties of a function before I’d started implementing.

I approached this by using a combination of example based (usually TDD) tests, the REPL for explorative testing and PBT.

My main issue with most of the example of PBT I’ve been through (and I must have tried at least 6 tutorials!) is that they are simple and algorithmic. By that I mean the functions were pure and tended to have properties that were easily verifiable and inputs that were easily generated. For example. testing sort on a vector or the behaviour of a queue.

I live in the world of business where most of my problems are about moving and transforming data. I don’t think I’ve implemented a sort or a queue like structure since I left university. My problems are messier and tend to involve inconvenient things like state.

Therefore I thought I would try and put together a simple but slightly more real world example to use PBT on that involved a RESTful API. I hope to show how and when I used PBT in combination with traditional REPL and example based tests.

My imaginary API is really simple. It consists of a ‘customers’ resource that will allow CRUD operations.