Parallel Tests

Node-tap includes the ability to run buffered child tests in parallel. There are two ways that this can be done: either via the command line interface, or within a single test program.

In both cases, you set a number of jobs that you want to allow it to run in parallel, and then any buffered tests are run in a pool which will execute that many test functions in parallel.

The default jobs value for the command line runner is equal to the number of CPUs on your system, so it's as parallel as makes sense. Within a single test file, the default jobs value is 1, because you rarely want to run the functions within a given suite in parallel.

Considerations for running parallel tests

The thing about running tests in parallel is that they can effectively run in any order, and at the same time.

That means that any test fixtures, ports, or files that a test writes must be created specially for that test, and not shared between tests. You cannot write tests that depend on being run in a specific order.

To help facilitate this, the process.env.TAP_CHILD_ID environment variable will be set to a number indicating which child process is currently being run. Instead of creating a folder called 'test-fixtures', you could create one called 'test-fixtures-' + process.env.TAP_CHILD_ID. Instead of spinning up a server on port 8000, you can have it listen on 8000 + (+process.env.TAP_CHILD_ID). (Note that environment variables are always strings, so we have to cast it to a number.)

This way, your tests will not collide with one another.

If you have some tests that must be order-dependent or share state, you can either put them all in the same test file, or in a folder containing a file named tap-parallel-not-ok, or turn off parallel tests by setting --jobs=1.

Parallel tests from the CLI

This is the simplest way to run parallel tests, and it happens by default.

In some reporters, it may seem like the output from each test file happens "all at once", when the test completes. That's because parallel tests are always buffered, so the command-line harness doesn't parse their output until they're fully complete. (Since many of them may be running at once, it would be very confusing otherwise.)

Newer test runners (those based on treport) show information about parallel tests as they are spawned.

Enabling/Disabling Parallelism in the test runner

If you set the --jobs option, then tests will be run in parallel by default.

However, you may want to have some tests run in parallel, and make others run serially.

To prevent any tests in a given folder from running in parallel, add a file to that directory named tap-parallel-not-ok. This will prevent tests from being run in parallel in that folder or any sub-folders.

To re-enable parallel tests in a given folder and its subfolders, create a file named tap-parallel-ok. This is only required if parallel tests had been disabled in a parent directory.

For example, if you had this folder structure:

test
├── parallel/
│   ├── all-my-ports-are-private-and-unique.js
│   ├── isolated.js
│   ├── no-external-deps.js
│   └── tap-parallel-ok
├── bar.js
├── foo.js
└── tap-parallel-not-ok

then running tap -j4 test/ would cause it to run bar.js and foo.js serially, and the contents of the test/parallel/ folder would be run in parallel.

As test files are updated to be parallel-friendly (ie, not listening on the same ports as any other tests, not depending on external filesystem stuff, and so on), then they can be moved into the parallel subfolder.

Parallel tests from the API

To run child tests in parallel, set t.jobs = <some number> in your test program. This can be set either on the root tap object, or on any child test.

The default number of jobs within a given test file is 1, regardless of what you specify on the command line.

If t.jobs is set to a number greater than 1, then tests will be run in buffered mode by default. To force a test to be serialized, set { buffered: false } in its options. You may also set TAP_BUFFER=0 in the environment to make tests non-buffered by default.

For example, imagine that you had some slow function that makes a network request or processes a file or something, and you want to call this function three times in your test program.

const t = require('tap')

t.test(function one (t) {
  someSlowFunction(function () {
    t.pass('one worked')
    t.end()
  })
})

t.test(function two (t) {
  someSlowFunction(function () {
    t.pass('two worked')
    t.end()
  })
})

t.test(function three (t) {
  someSlowFunction(function () {
    t.pass('three worked')
    t.end()
  })
})

That produces this output:

TAP version 13
# Subtest: one
    ok 1 - one worked
    1..1
ok 1 - one # time=283.987ms

# Subtest: two
    ok 1 - two worked
    1..1
ok 2 - two # time=352.492ms

# Subtest: three
    ok 1 - three worked
    1..1
ok 3 - three # time=313.015ms

1..3
# time=957.096ms

If we update our test function to add t.jobs = 3, then the output looks like this instead:

TAP version 13
ok 1 - one # time=215.87ms {
    ok 1 - one worked
    1..1
}

ok 2 - two # time=97.694ms {
    ok 1 - two worked
    1..1
}

ok 3 - three # time=374.099ms {
    ok 1 - three worked
    1..1
}

1..3
# time=382.468ms

Each test still takes a few hundred ms, but the overall time is way down. Also, they're using the more streamlined buffered subtest style, so that they can run in parallel.

Caveats about Parallel Tests

Parallelism is not a magic sauce that makes everything go fast.

It's a good fit when your test spends a lot of time waiting in sleep mode (for example, waiting for I/O to complete), or if you have lots of CPUs on your computer. But if your tests are on-CPU most of the time, then there's often little benefit to running more of them than you have CPUs available.

Parallel testing also means that your tests have to be written in an independent way. They can't depend on being run in a given order, which means it's a bad idea to have them share pretty much any state at all.