Skip to main content

What's new in fast-check 4.9.0?

· 13 min read
Nicolas Dubien
Nicolas Dubien
fast-check maintainer

Supporting shrink is crucial for the built-in arbitraries in fast-check. For this release, we deeply reworked entityGraph to give it built-in shrinking support. But this release is above all about performance. We spent the last few weeks tracking any optimization that we could put into fast-check on its critical and hot code paths.

Continue reading to explore the detailed updates it brings.

Rebuilding entityGraph on core primitives​

Our initial implementation of entityGraph was pretty ad-hoc. It was mostly a manual implementation of an Arbitrary barely re-using any existing primitives from fast-check. It worked but missed key capabilities coming with our core building blocks.

The reason for that choice is twofold:

  • The arbitrary was complex and it took us time to get a comprehensive enough picture of how we would implement it. As such going manual helped us sketch how we could generate such values. This first sketch got translated in this release into core building blocks.
  • We were missing a key building block: chainUntil (landed in 4.8.0).

In this minor release, we switched from a manual and tailored implementation of the Arbitrary class to an arbitrary composing several building blocks from fast-check. This switch unlocked several nice things:

  • The arbitrary now benefits from shrinking capabilities. We don't have to do special tricks for it to work.
  • The arbitrary also benefits from all performance optimizations made to core building blocks. As such optimizing them benefits other arbitraries even more.

Performance at heart​

Aiming for performance has always been in our DNA. We want the cost of running tests with fast-check to be as little as possible. As such this release focused on finding things we could make faster to benefit hot code paths as much as possible.

Strategy​

We sat down and thought about what should and what should not be optimized. If you think of one of your property-based snippets, you'll probably quickly reach this conclusion:

  • Instantiating an instance of Arbitrary is done once per test
  • Pulling and generating values out of an Arbitrary is achieved a hundred times per test (by default)
  • Reducing to smaller values is almost never done, as it only happens when there are bugs

Put differently, making the generate code path faster at the cost of a slower shrink is no problem. The same holds when trading faster generation for slower initialization.

Process​

Earlier this year we had the privilege to be accepted as part of the Claude for Open Source licensing. We wanted to see and try to make it capable of helping us to track down slow code paths and propose optimizations for them.

We decided to tell Claude how performance troubleshooting works, what it can usually look for in terms of optimizations... To achieve that we drafted a CLAUDE.md summarizing our mission. The file was split into several sections. The following list gives you a quick highlight of the key principles we used for each of them.

  1. How to find slow code paths?

    Write down benchmarks, run them and use profiling tools to extract the worst offenders. It's better to optimize code snippets accountable for huge parts of the runtime cost. Node comes with an extra toolset to find out deoptimizations, use dexnode to record them.

  2. What are the most common optimization tricks in JavaScript and also generally?

    Build a set of common optimization tricks that are worth considering when optimizing code. To construct that set:

    • Scan all the PRs related to performance (⚡️) that we merged over time in both dubzzz/fast-check and dubzzz/pure-rand
    • Read through Marvin Hagemeister's blog posts
    • Enrich with your personal knowledge and extra resources that you could find online on that matter

  3. How to confirm an optimization works?

    Run the code, benchmark it, profile it. Make sure that the place you optimized really shrunk in profiling. Beware of micro-benchmarks. When writing benchmarks always consider several possible entry sets and don't just focus on one of them in your run.

  4. What is important to optimize in fast-check?

    Well, this is just more or less about the Strategy section above.

  5. How to proceed?

    Use worktrees. Run one dedicated sub-agent per optimization. Only consider opening a PR if the optimization is really impressive or useful. Share benchmark snippets and benchmark results as part of the PR description. Always re-confirm the benchmark results.

The results​

Claude found out interesting places to look at and proposed some useful tricks.

But everything had to be carefully re-assessed. Some optimizations were incorrect or not saving time on useful parts of the code. Some were making the code hard to read or the bundle 10x larger. Regarding the places it suggested to optimize, I think this is where it shone: the proposed places were always interesting to consider. Some were useless but at least they opened my eyes to parts where we could eventually move faster.

The combination of LLM-guided performance review and human work proved efficient for the library as a whole.

Running a dummy property such as:

fc.assert(fc.property(fc.constant(1), (_c) => {}));

Will be +50% faster with 4.9.0.

But we not only improved the basic runtime of an empty property. We also improved most of our arbitraries. The following table summarizes the measurements that we made when we compared 4.8.0 against 4.9.0:

Benchmark4.8.0 ops/s4.9.0* ops/sChange
stringMatching(/^abc$/)1,97434,751+1661%
memo(tree)309939+204%
letrec(tree)314785+150%
record({ a: integer() })1,0312,406+133%
tuple(integer())3,8315,448+42%
integer()9,10412,498+37%
integer().map(value+1)7,0989,478+34%
stringMatching(/^[a-zA-Z0-9]+$/)464619+33%
maxSafeInteger()6,6688,835+33%
integer().filter(true)8,33911,024+32%
oneof(integer(), integer())5,1966,469+25%
oneof({ weight, arbitrary }, …)5,2626,482+23%
ipV4()1,1231,377+23%
array(integer())1,4631,780+22%
float()4,0754,948+21%
option(integer())5,1876,235+20%
date()3,0823,660+19%
array(integer(), max 100)252294+17%
webUrl()133155+17%
ipV6()189220+16%
emailAddress()7181+15%
uniqueArray(integer())1,0431,191+14%
anything()5360+14%
boolean()11,78713,383+14%
map(string(), integer())125139+11% noise
integer().chain(integer())3,6013,988+11%
json()5054+9%
set(integer())801873+9%
base64String()890968+9% noise
dictionary(string(), integer())7177+8%
string()9351,002+7%
uuid()552587+6%
mixedCase(string())463489+6% noise
string({ unit: 'grapheme' })647682+5% noise
constantFrom(1, 2)23,29823,934+3% noise
constant(1)34,47535,099+2% noise
string(max 100)153154+1% noise
bigInt()969974+1% noise
subarray([1, 2, 3, 4, 5])1,8811,888+0% noise
double()1,6011,600−0% noise
entityGraph(employee → manager? + team)11.310.8−4%

Measured on Node v22.22.2 running fast-check's own arbitraries.bench.ts: fast-check@4.8.0 with pure-rand@8.4.0 vs fast-check@4.9.0 with pure-rand@8.4.2.

Changelog since 4.8.0​

The version 4.9.0 is based on version 4.8.0.

Features​

  • (PR#7008) Towards shrinkable entityGraph thanks to chainUntil

Fixes​

  • (PR#7010) Bug: Fix latent state-sharing bug in entityGraph
  • (PR#7063) Bug: Equiprobable alternatives in stringMatching
  • (PR#6973) CI: Drop caches on push for build package flow
  • (PR#6971) CI: Only mark fast-check's releases as latest
  • (PR#6974) CI: Drop pull_request_target flows
  • (PR#6975) CI: Drop discussion creation on release publish
  • (PR#6976) CI: Drop caches from publication steps
  • (PR#6977) CI: Revert "Drop caches from publication steps"
  • (PR#6978) CI: Make zizmor audit a required status check
  • (PR#6991) CI: Remove Claude Code workflow
  • (PR#6994) CI: Replace pnpm dlx with pnpm exec for pkg-pr-new
  • (PR#6995) CI: Inline zizmor ignores in workflow
  • (PR#6996) CI: Move to devEngines.packageManager
  • (PR#7005) CI: Update PULL_REQUEST_TEMPLATE.md
  • (PR#7011) CI: Drop OTP prompt from npm publish
  • (PR#7013) CI: Switch release jobs to npm stage publish
  • (PR#7027) CI: Run benchmarks against main
  • (PR#7037) CI: Use comparison mode for bench
  • (PR#7069) CI: Run pnpm dedupe to deduplicate lockfile
  • (PR#6959) CI: Announce releases on Bluesky
  • (PR#7105) CI: Switch to actions/attest for attestations
  • (PR#7117) CI: Use pnpm version in changelog script
  • (PR#7120) CI: Allow unclean tree in changelog generation
  • (PR#7125) CI: Stage publish using pnpm in publish jobs
  • (PR#7065) Clean: Delete skills directory
  • (PR#6983) Doc: Tweak PR Template to hint AI agents into revealing themselves
  • (PR#7092) Doc: Add back skills directory
  • (PR#7095) Doc: Add release notes for fast-check 4.8.0
  • (PR#7104) Doc: Add makeeno as doc contributor
  • (PR#7103) Doc: Fix info box in docs
  • (PR#7108) Doc: Add jneidel as doc contributor
  • (PR#7035) Performance: Faster fc.integer on generate
  • (PR#7046) Performance: Faster fc.record on generate
  • (PR#7047) Performance: Faster fc.dictionary on generate
  • (PR#7048) Performance: Faster fc.webPath/fc.webUrl on generate
  • (PR#7050) Performance: Faster fc.stringMatching for \W \D \S .
  • (PR#7054) Performance: Faster fc.stringMatching on generate
  • (PR#7049) Performance: Drop nested tuple on generate for fc.record
  • (PR#7045) Performance: Faster fc.entityGraph on generate
  • (PR#7071) Performance: Early exit on empty tuple in fc.entityGraph
  • (PR#7004) Refactor: Extract code from onTheFlyLinksForEntityGraph
  • (PR#7006) Refactor: Move generate logic to the Arbitrary for entityGraph
  • (PR#7007) Refactor: Introduce ProductionState for onTheFlyLinks...
  • (PR#6990) Script: Skip scripts during pnpm i in changelog generation
  • (PR#7039) Script: More benchmark commands
  • (PR#6972) Security: Pass --ignore-scripts to pnpm i calls
  • (PR#7028) Test: Add benchmarks for key arbitraries
  • (PR#7034) Test: Expand benchmark coverage across arbitrary families
  • (PR#7041) Test: Add runners related benchs
  • (PR#7064) Test: Clarify arbitrary benchmark names
  • (PR#7068) Test: More reliable arbitraries.bench.ts
  • (PR#7088) Typo: Typo in type EntityGraphContraints