Roman's blog

Best node.js test framework, with benchmarks

Roman's photo
Roman
·

11 min read

Table of contents

According to npm trends, Jest is by far the most popular test framework nowadays, Mocha has 3 times less weekly downloads, and Vitest has 5 times less.

I'm developing a library that has lots of tests (> 2.5k) run by Jest, and I decided to find out if there is a better alternative to run tests faster.

I benchmarked test execution with Jest, Vitest, Bun, Mocha, Node's native test runner, Tapjs, Ava, Japa.

Along the way I encountered some surprises and learned something new, I'll gladly share it with you in this post. (benchmarks results are closer to the bottom).

Test frameworks overview

I won't go into details on each one, but going to answer these questions:

  • easiness of setup (for TypeScript)

  • documentation quality

  • concurrent tests support

  • ESM support

By ESM support I mean that it should work out of the box, or at least it must not require endless hours trying to make it work. Officially, none of the tools admits a lack of ESM support.

To check whether the framework has ESM support or not, I import an ESM-only package (such as p-queue), and a test crashes when no ESM.

Concurrent tests

This is what I learned about earlier today while was coding the benchmarks, and I'm going to use it in the future to speed up tests.

Typically, the test runner spawns several workers and runs your test files in parallel. But tests inside those files are still executed sequentially - one by one.

This means that if you have 100 tests in the same file each waiting for 10ms, the whole suite will take 10 seconds.

But one of the runners does it differently by default and runs all tests concurrently (as in Promise.all), it processed async tests in 1.5s while others were taking 10+.

The difference is huge, how is it possible that only one tool had such a bright idea, but more popular tools cannot do that?

It turned out, you can also do that with Jest and Vitest! This is possible, though it is not enabled by default, and for good reasons.

(link to Jest test.concurrent docs, Vitest test.concurrent docs)

There are good chances that concurrent tests will fail because of race conditions. Concurrent tests are a powerful feature that can save a lot of execution time, but they should be applied with caution, and won't fit any use-case, that's why it is not a default behavior.

Jest

Setup takes some time but is pretty simple. Two packages for jest:

npm i -D jest @types/jest

For TS support, I've used @swc/jest and esbuild-jest.

jest.config.cjs config file:

module.exports = {
  transform: {
    "^.+\\.(t|j)sx?$": "@swc/jest", // or "esbuild-jest"
  },
};

Then add a "test" script into package.json. I prefer to have it in watch mode by default, and a separate script to run without watching. Also to lower verbosity level by default.

{
  "scripts": {
    "test": "jest --watch --verbose false",
    "check": "jest",
  }
}

Run it with:

# test some file in a watch mode
npm t someFile

# run all tests
npm run check

It seems pretty quick and straightforward, but neither swc nor esbuild is mentioned in the official documentation, which is misleading people into thinking that Jest is very slow by itself.

  • As mentioned above, the setup is not straightforward, docs could be better.

  • No ESM support

  • Has concurrent tests, but they are experimental and have some gotchas related to beforeEach/afterEach behavior.

Vitest

Vitest is wonderful, works out of the box, requires minimal effort when migrating from Jest, documentation is excellent.

  • Setup is effortless.

  • Docs are visually pleasant, well structured, and have all that's needed.

  • Designed for ESM.

  • Concurrent tests, as I can tell from docs, are better thought-out than in Jest. Has describe.concurrent, unlike Jest.

Bun

This post is entitled "Best node.js framework", but, as you might have guessed, Bun isn't one. But would you care about such nuances if the tests of your node.js application would run 10 times faster? As for me, I'd be happy to use Bun as a test framework as long as it brings practical benefits.

  • Setup is effortless.

  • Good docs.

  • ESM works perfectly fine.

  • No concurrent tests.

  • Most importantly, all test files are served by a single thread.

The last point reveals that you shouldn't expect good performance from Bun on async tests, which is disappointing. Let's hope that Bun will catch up on this soon, I'll be waiting for it and wishing luck to Bun's team.

Mocha

Mocha is a lightweight test runner, typically used with Chai for "expect" functionality and Sinon for stubs.

For TS support, I installed esbuild-register package, then created such config .mocharc.json:

{
  "extension": ["ts"],
  "require": "esbuild-register"
}

And running test with the command:

mocha -r esbuild-register --parallel src/mocha/sync/*.test.ts

  • Setup isn't straightforward.

  • Docs have lots of text but aren't well organized.

  • Requires a full test path with wildcard, it's not as nice as in Jest or Vitest.

  • ESM works fine.

  • Does not support concurrent tests.

node.js native test runner

Node.js offers a lightweight test runner out of the box. It is the most minimalistic one.

Docs do not mention TypeScript at all, I figured out that it can be launched with esbuild-register in the same way as for Mocha:

node -r esbuild-register --test src/node/sync/*.test.ts

  • Setup isn't straightforward, it took me a while to utilize the esbuild-register.

  • All documentation is on a single page, which is not perfect but is convenient enough.

  • Requires a full test path, the same as in Mocha.

  • ESM works fine.

  • No concurrent tests.

Tapjs

Tap has an unusual syntax, for example:

t.test("test", (t) => {
  t.equal(2 + 2, 4);
  t.end(); // why?
});

Unlike other tools, Tap has an interactive REPL, watching tests happens here.

It has a lot of plugins for various cases, such as for snapshot testing.

It's shipped with ts-node/esm out of the box and can run typescript tests in an ESM project.

  • Effortless setup as in Vitest.

  • Docs look old-school but are structured and complete. Though, it doesn't explain why t.end() is needed.

  • ESM is supported.

  • No concurrent tests.

Ava

Ava is more focused on asynchronous tests, runs all tests concurrently by default, has various methods to define test execution orders, and has various plugins.

To enable TS, I installed tsimp package as suggested in the documentation, and added this config to package.json:

{
  "ava": {
    "extensions": {
      "ts": "module"
    },
    "nodeArguments": [
      "--import=tsimp"
    ]
  }
}

swc and esbuild don't seem to be supported by Ava, but on the other hand, with the tsimp suggested by Ava tests are type-checked. Ava compiles tests to a local directory to speed things up on the following runs.

  • TypeScript requires extra steps, but overall, setup is simple.

  • Docs are in GitHub in md files, which is very inconvenient.

  • ESM does not work. Searching in docs didn't give anything useful.

  • Concurrent test are by default.

Japa

Japa's is designed as a lightweight replacement for Jest/Vitest.

I encountered various issues while setting it up, so it doesn't seem to be ready yet.

Japa is a library, you define your own bin/test.ts file where you call it. This is a good idea but it takes a bit more time to set up than other tools.

I couldn't find how to run tests under a certain path, which was possible with every other test runner. It's doable because Japa is a library and is called from your own script, but it's not done out of the box and you'd need to handle the CLI arg manually.

  • Setup took more time than expected, there are some rough edges, but wasn't a problem.

  • Docs look very nice, no problems.

  • ESM is supported.

  • No concurrent tests.

Benchmarks

Some tools are more performant at handling synchronous tests, others are better at asynchronous, so I've made 2 kinds of tests:

  • sync test is as simple as 2 + 2 = 4, with the test block repeated 1000 times in a loop.

  • async test awaits a setTimeout(10), also repeated 1000 times.

I copy-pasted both sync and async suites 4 times each to see if the test runner can run multiple suites in parallel.

Let me know in the comments if you'd like to read benchmark sources and I'll push it to github and attach a link.

Synchronous tests

bun:                █ 0.08s
mocha with esbuild: ████ 0.43s
japa tsx:           ██████ 0.6s
node esbuild:       ████████ 0.85s
vitest:             █████████ 0.9s
node tsx:           █████████ 0.9s
jest swc:           ████████████ 1.2s
jest esbuild:       ████████████ 1.27s
tap esbuild:        ████████████████████████████ 2.94s
ava:                ███████████████████████████████████ 3.62s

Bun, as expected, is crazy. Mocha is a good pick for synchronous tests. Vitest's website says "it is fast", and Japa's website says it's "faster", both have kept their promise.

Node's test runner, despite being the most minimalistic, couldn't surpass Mocha.

tsx is a wrapper on top of esbuild, no surprise that it takes a bit longer for tsx.

Tap and Ava are at the bottom because they both are shipped with not very performant TypeScript parsers and don't support running them with swc or esbuild.

Asynchronous tests

ava:                 █ 1.29s
vitest:              ██ 2.75s
jest with swc:       ██ 2.85s
jest with esbuild:   ██ 2.98s
node with esbuild:   ███████████ 11.13s
node with tsx:       ███████████ 11.25s
mocha with esbuild:  ███████████ 11.34s
japa with tsx:       ███████████ 11.4s
tap:                 ██████████████ 14.14s
bun:                 ███ ...a few moments later... ███ 41.36s

Ava nails it with its optimizations for async tests.

I used test.concurrent for Vitest and Jest, that's why they both perform much faster than the others below.

Node's test runner, Mocha, Japa, Tap don't support concurrent tests so they wait all setTimeout(10) * 1000 sequentially, which results in 10s, and add overhead on top of that.

Tap is less performant than others in this category, it's shipped with ts-node/esm which also adds overhead.

There were 4 test files, 1000 tests in each, every test waits for a 10ms timeout. All contenders above are running the test files in parallel, except for Bun.

Bun is out of consideration for async tests yet. Running test files in parallel is an essential feature, and it's a pity that Bun is missing it.

So which one is the best?

I guess this is a common case, most of my tests are synchronous, less are asynchronous but they're taking more time to finish. So async results are more important (no to Bun and to Tap), but sync results are also impactful (no to Ava and Tap). Ava and Tap also didn't make a good impression on their easiness of use and docs. Japa needs more time to stabilize. Node's test runner with its focus on minimalism is probably the last one I'd pick. Mocha looks like a fine choice: 2nd fast at sync tests, having concurrent tests isn't critical because they're brittle, but it requires additional setup, searching "how do to X, Y, Z in Mocha" when these are out-of-box features in others, and remembering a slightly different interface.

Vitest appears to be the one! It's the easiest to set up, has the best docs, is actively maintained, has more features, and is very fast.

Don't trust benchmarks, or Jest vs Vitest

In the benchmarks above, Vitest is a little bit faster than Jest. It's not enough to switch, but I tried to switch anyway, out of curiosity to find out what the difference will be on a real project.

In my library with thousands of tests Vitest performs ~2x slower than Jest.

Why was Vitest better at fake tests? Because there were only 4 files. I created another 196 fake test files for the benchmark, and Vitest became 2-3x slower on fake tests as well.

But then I've found a savior config option: set isolate to false, so that the test workers will share their memory, if you're unlucky the tests won't work, but if you're lucky it is much more efficient.

In my library, Vitest with isolate: false performed a little slower than Jest, but some of the tests became broken by it. Similar to test.concurrent, this option gives a performance boost but is disabled by default for a reason.

Also, I find Jest's test output to be better. Vitest's output is more colorful and looks better designed, but it's annoyingly flickering when running lots of tests, Jest doesn't have this problem.

Conclusion

If you don't need or want ESM, Jest is the best, it is fast if you aren't following its official docs, is easy to use, and has a decent feature set.

If you need or want ESM, Vitest is the best.

Can do ESM, runs faster, but is less convenient - Mocha.

Want to write a quick test without installing and configuring a test runner? Bun will save time on this.

Cannot trust any third-party libraries including Bun, or simply running out of space? Try the node's native test runner.

Ava is for the best performance of concurrent tests. But has the worst documentation, IMO.

Japa has to stabilize more. As for Tap, created 13 years ago, it is still actively maintained, its users probably have their reasons, but I couldn't see a reason to choose Tap.

Node.jsWeb DevelopmentTesting