Node-tap includes the ability to run buffered child tests in parallel. There are two ways that this can be done: either via the command line interface, or within a single test program.
In both cases, you set a number of jobs
that you want to allow it to
run in parallel, and then any buffered tests are run in a pool which
will execute that many test functions in parallel.
The default jobs
value for the command line runner is equal to the number
of CPUs on your system, so it's as parallel as makes sense. Within a
single test file, the default jobs
value is 1
, because you rarely want
to run the functions within a given suite in parallel.
The thing about running tests in parallel is that they can effectively run in any order, and at the same time.
That means that any test fixtures, ports, or files that a test writes must be created specially for that test, and not shared between tests. You cannot write tests that depend on being run in a specific order.
To help facilitate this, the process.env.TAP_CHILD_ID
environment
variable will be set to a number indicating which child process is
currently being run. Instead of creating a folder called 'test-fixtures'
,
you could create one called 'test-fixtures-' + process.env.TAP_CHILD_ID
.
Instead of spinning up a server on port 8000
, you can have it listen on
8000 + (+process.env.TAP_CHILD_ID)
. (Note that environment variables are
always strings, so we have to cast it to a number.)
This way, your tests will not collide with one another.
If you have some tests that must be order-dependent or share state, you can
either put them all in the same test file, or in a folder containing a file
named tap-parallel-not-ok
, or turn off parallel tests by setting
--jobs=1
.
This is the simplest way to run parallel tests, and it happens by default.
In some reporters, it may seem like the output from each test file happens "all at once", when the test completes. That's because parallel tests are always buffered, so the command-line harness doesn't parse their output until they're fully complete. (Since many of them may be running at once, it would be very confusing otherwise.)
Newer test runners (those based on treport) show information about parallel tests as they are spawned.
If you set the --jobs
option, then tests will be run in parallel by
default.
However, you may want to have some tests run in parallel, and make others run serially.
To prevent any tests in a given folder from running in parallel, add a
file to that directory named tap-parallel-not-ok
. This will prevent
tests from being run in parallel in that folder or any sub-folders.
To re-enable parallel tests in a given folder and its subfolders,
create a file named tap-parallel-ok
. This is only required if
parallel tests had been disabled in a parent directory.
For example, if you had this folder structure:
test
├── parallel/
│ ├── all-my-ports-are-private-and-unique.js
│ ├── isolated.js
│ ├── no-external-deps.js
│ └── tap-parallel-ok
├── bar.js
├── foo.js
└── tap-parallel-not-ok
then running tap -j4 test/
would cause it to run bar.js
and
foo.js
serially, and the contents of the test/parallel/
folder
would be run in parallel.
As test files are updated to be parallel-friendly (ie, not listening
on the same ports as any other tests, not depending on external
filesystem stuff, and so on), then they can be moved into the
parallel
subfolder.
To run child tests in parallel, set t.jobs = <some number>
in your
test program. This can be set either on the root tap object, or on
any child test.
The default number of jobs within a given test file is 1, regardless of what you specify on the command line.
If t.jobs
is set to a number greater than 1, then tests will be run
in buffered
mode by default. To force a test to be serialized, set
{ buffered: false }
in its options. You may also set
TAP_BUFFER=0
in the environment to make tests non-buffered by
default.
For example, imagine that you had some slow function that makes a network request or processes a file or something, and you want to call this function three times in your test program.
const t = require('tap')
t.test(function one (t) {
someSlowFunction(function () {
t.pass('one worked')
t.end()
})
})
t.test(function two (t) {
someSlowFunction(function () {
t.pass('two worked')
t.end()
})
})
t.test(function three (t) {
someSlowFunction(function () {
t.pass('three worked')
t.end()
})
})
That produces this output:
TAP version 13
# Subtest: one
ok 1 - one worked
1..1
ok 1 - one # time=283.987ms
# Subtest: two
ok 1 - two worked
1..1
ok 2 - two # time=352.492ms
# Subtest: three
ok 1 - three worked
1..1
ok 3 - three # time=313.015ms
1..3
# time=957.096ms
If we update our test function to add t.jobs = 3
, then the output
looks like this instead:
TAP version 13
ok 1 - one # time=215.87ms {
ok 1 - one worked
1..1
}
ok 2 - two # time=97.694ms {
ok 1 - two worked
1..1
}
ok 3 - three # time=374.099ms {
ok 1 - three worked
1..1
}
1..3
# time=382.468ms
Each test still takes a few hundred ms, but the overall time is way
down. Also, they're using the more streamlined buffered
subtest
style, so that they can run in parallel.
Parallelism is not a magic sauce that makes everything go fast.
It's a good fit when your test spends a lot of time waiting in sleep mode (for example, waiting for I/O to complete), or if you have lots of CPUs on your computer. But if your tests are on-CPU most of the time, then there's often little benefit to running more of them than you have CPUs available.
Parallel testing also means that your tests have to be written in an independent way. They can't depend on being run in a given order, which means it's a bad idea to have them share pretty much any state at all.