Custom reports

Customize how to report failures.

Default Report

When failing assert automatically format the errors for you, with something like:

**FAIL**  sort.test.mjs > should sort numeric elements from the smallest to the largest one
Error: Property failed after 1 tests
{ seed: -1819918769, path: "0:...:3", endOnFailure: true }
Counterexample: [[2,1000000000]]
Shrunk 66 time(s)
Got error: AssertionError: expected 1000000000 to be less than or equal to 2

While easily redeable, you may want to format it differently. Explaining how you can do that is the aim of this page.

How to read such reports?

If you want to know more concerning how to read such reports, you may refer to the Read Test Reports section of our Quick Start tutorial.

Verbosity

The simplest and built-in way to change how to format the errors in a different way is verbosity. Verbosity can be either 0, 1 or 2 and is defaulted to 1. It can be changed at assert's level, by passing the option verbose: <your-value> to it.

You may refer to Read Test Reports for more details on it.

New Reporter

In some cases you might be interested into fully customizing, extending or even changing what should be a failure or how it should be formated. You can define your own reporting strategy by passing a custom reporter to assert as follow:

fc.assert(
  // You can either use it with `fc.property`
  // or `fc.asyncProperty`
  fc.property(...),
  {
    reporter(out) {
      // Let's say we want to re-create the default reporter of `assert`
      if (out.failed) {
        // `defaultReportMessage` is an utility that make you able to have the exact
        // same report as the one that would have been generated by `assert`
        throw new Error(fc.defaultReportMessage(out));
      }
    }
  }
)

In case your reporter is relying on asynchronous code, you can specify it by setting asyncReporter instead of reporter. Contrary to reporter that will be used for both synchronous and asynchronous properties, asyncReporter is forbidden for synchronous properties and makes them throw.

Before reporter and asyncReporter

In the past, writing your own reporter would have been done as follow:

const throwIfFailed = (out) => {
  if (out.failed) {
    throw new Error(fc.defaultReportMessage(out));
  }
};
const myCustomAssert = (property, parameters) => {
  const out = fc.check(property, parameters);

  if (property.isAsync()) {
    return out.then((runDetails) => {
      throwIfFailed(runDetails);
    });
  }
  throwIfFailed(out);
};

CodeSandbox Reporter

In some situations, it can be useful to directly publish a minimal reproduction of an issue in order to be able to play with it. Custom reporters can be used to provide such capabilities.

For instance, you can automatically generate CodeSandbox environments in case of failed property with the snippet below:

import { getParameters } from 'codesandbox/lib/api/define';

const buildCodeSandboxReporter = (createFiles) => {
  return function reporter(runDetails) {
    if (!runDetails.failed) {
      return;
    }
    const counterexample = runDetails.counterexample;
    const originalErrorMessage = fc.defaultReportMessage(runDetails);
    if (counterexample === undefined) {
      throw new Error(originalErrorMessage);
    }
    const files = {
      ...createFiles(counterexample),
      'counterexample.js': {
        content: `export const counterexample = ${fc.stringify(counterexample)}`
      },
      'report.txt': {
        content: originalErrorMessage
      }
    }
    const url = `https://codesandbox.io/api/v1/sandboxes/define?parameters=${getParameters({ files })}`;
    throw new Error(`${originalErrorMessage}\n\nPlay with the failure here: ${url}`);
  }
}

fc.assert(
  fc.property(...),
  {
    reporter: buildCodeSandboxReporter(counterexample => ({
      'index.js': {
        content: 'console.log("Code to reproduce the issue")'
      }
    }))
  }
)

CodeSandbox documentation

The official documentation explaining how to build CodeSandbox environments from an url is available here: https://codesandbox.io/docs/importing#get-request.

Customize toString

By default, fast-check serializes generated values using its internal stringify helper. Sometimes you may want a better stringified representation of your instances. In such cases, you have several solutions:

1. If your instance defines a toString method, it will used to properly report it, unless you've defined one of the following methods, which take precedence.
2. If defining toString method is to intrusive, you can use toStringMethod and asyncToStringMethod.

In most cases, toStringMethod is sufficient. This is the serializer method that fast-check uses to serialize your instance in any context: synchronous or asynchronous.

Object.defineProperties(myInstanceWithoutCustomToString, {
  [fc.toStringMethod]: { value: () => 'my-value' },
});
// here your instance defines a custom serializer
// that will be used by fast-check whenever needed

However, if you're working with asynchronous values, you may need an async method to retrieve the value. For example:

Object.defineProperties(myPromisePossiblyResolved, {
  [fc.asyncToStringMethod]: {
    value: async () => {
      const resolved = await myPromisePossiblyResolved;
      return `My value: ${resolved}`;
    },
  },
});

Limitations of async variant

Note that:

• asyncToStringMethod is only used for asynchronous properties.
• Although asyncToStringMethod is marked as asynchronous, it should resolve almost instantly.

Test your custom toString

One way to ensure that your instances will be properly stringified is to call the stringify function provided by fast-check. This will give you a preview of how your instances will be represented in the output.