Some ScalaCheck generators fail too much

2015-12-29T00:00:00Z

Most of the code I’ll be working with in my new job (BTW blog: I have a new job) is written in Scala and uses property based testing with ScalaCheck. Yesterday I ran into a problem with an existing test suite that suddenly began failing with too many discarded tests:

[info] FormattersSpec
[info]   Formatters are invertible for:
[info]     + Mapping
[info]     + Identifier
[info]
[error]     x Metadata
[error]  Gave up after only 39 passed tests. 197 tests were discarded. (FormattersSpec.scala:11)

This test generates random Metadata values and makes sure that they can be serialised and deserialised correctly (i.e. values can be round-tripped). The property being test here is identical, only the Arbitrary, Serialise, and Deserialise instances vary in each case. The truly odd thing is that the pertinent code looks like this:

case class Identifier(name: String)
case class Metadata (id: Identifier, maps: Set[Identifier])

implicit val ArbIdentifier = Arbitrary(
  for {
    name  <- arbitrary[String]
  } yield Identifier(name)
)

implicit val ArbMetadata = Arbitrary(
  for {
    identifier <- arbitrary[Identifier]
    mappings   <- arbitrary[Set[Identifier]]
  } yield Metadata(identifier, mappings)
)

My first step was redefining a few related Arbitrary instances to avoid using suchThat (which discards invalid values) but this didn’t fix the problem. Eventually I tried redefining ArbMetadata like this:

implicit val ArbMetadata = Arbitrary(
  for {
    identifier <- arbitrary[Identifier]
    mappings   <- Gen.const(Set.empty[Set[Identifier]])
  } yield Metadata(identifier, mappings)
)

and the problem went away. Trying to use arbitrary[Set[Identifier]] in various ways in the Scala REPL confirmed that it is the problem; we can easily generate as large a List[Identifier] as we like, but a Set[Identifier] fails fairly frequently:

// This always generates a Some[List[Identifier]] value.
Gen.listOfN(100, arbitrary[Identifier]).map(_.length).sample
// Sometimes we get a Some[List[Set[Identifier]]] and others None.
Gen.listOfN(100, arbitrary[Set[Identifier]]).map(_.length).sample

It appears as though whatever mechanism is used by arbitrary[Set[_]] to construct the sets, it doesn’t fails when the generator for the value type returns duplicate elements. You can confirm this easily by trying arbitrary[Set[Unit]]; any Gen[Unit] has no choice but to return a the single value of type Unit (or to fail) and, as expected, this almost never succeeds. Replacing the problematic arbitrary[Set[Identifier]] in the original code with arbitrary[Seq[Identifier]].map(_.toSet) resolves the issue: constructing a set from a list of possibly duplicate Identifiers always works.

After a bit of reading in the ScalaCheck source code it seems as though the root cause of this problem is some instance of CanBuildFrom[Set[_], A, Set[A]] but I’ve no idea how to go about figure out which one or why it’s broken. In any case, I now know a bit more about working with Scala.

For more information, see the ScalaCheck issue #89.

Passing Curiosity: Posts tagged properties

Some ScalaCheck generators fail too much