Node TAP 21.1.1

Changelog

21.1#

21.0#

20.0#

19.2#

19.1#

19.0#

18.8#

18.7#

18.6#

18.5#

18.4#

Note: The formatting of Symbol objects and enumerable Symbol properties has changed in snapshots and object diffs.

You may need to run tap --snapshot after updating to this release, and/or update some tests that were relying on symbol property keys being ignored.

18.3#

18.2#

18.1#

This release adds much better support for interoperating with the Node.js built-in test runner and framework.

When tap is run with NODE_TEST_CONTEXT=child-v8 in the environment, it will output a stream of serialized test messages, rather than TAP output.

Features#

Fixes#

18.0#

Major rewrite.

Highlights#

CLI#

Instead of always treating positional arguments as test names, the tap CLI now takes a subcommand as the first argument. If a known subcommand is not the first argument, or if no subcommand is provided, then it is treated as tap run ....

The tap plugin [add|rm|list] command is added.

The tap repl command is added, replacing tap --watch. Run tap repl help for a list of repl commands.

The tap replay command is added, which replays the results of the previous test run without running the tests again.

The tap report command is added, which runs a coverage report from the most recent test run.

The tap list command is added, which will print a list of test files that will be run.

The tap config [list [<key>] | edit | dump] command is added, taking the place of the --dump-config flag from previous versions. This makes it much easier to inspect how and where configurations are set, and edit them as needed.

The tap-parallel-ok and tap-parallel-not-ok files are no longer relevant. Instead, the --serial config can be used to specify folders where tests may not be run in parallel. Otherwise, tests are always run in parallel.

Tap no longer automatically reports coverage to coveralls when a COVERALLS_REPO_TOKEN is present in the environment.

Wherever possible, noise has been reduced, and useful information is surfaced. The goal is to show as little information as is helpful, and no less.

Plugins#

Almost all functionality previously provided in the Test class directly is now offloaded to plugins, which can be added or removed by use of the tap plugin command or the plugin configuration option.

Plugins are modules that at least one of the export the following fields:

A loader, which will be added to the set of loaders used to execute test files.

A config object, to add configuration fields that tap knows how to handle.

A list of testFileExtensions, adding to the set that tap will load by default.

A plugin method, which takes a Test object and its options as an argument, and can return any object. Whatever methods and properties are on the returned object will be present on the corresponding Test object used in tests.

Configuration#

Configuration has been rewritten for greater strictness, reliability, and extensibility, but some things that used to work no longer do.

All configuration fields are exported to the environment in UPPER_SNAKE case, with the prefix TAP_. Options that can take multiple values are delimited by a \n character.

The special extends option can be used in a .taprc or the tap section of a package.json file to inherit configuration from another file or a dependency package. The --rcfile option is removed, since it would be extraneous. To use a different config file, put something like this in your package.json file:

{
  "tap": {
    "extends": "./my-tap-configs.yaml"
  }
}

The test-ignore option is replaced with exclude, and is now a glob rather than a regular expression. The exclude option can be specified multiple times on the command line, or as an array of strings in a config file.

The test-regexp string option is replaced with the include option, and is now a glob rather than a regular expression. The include option can be specified multiple times on the command line, or as an array of strings in a config file. The include option may also contain the special token __EXTENSIONS__ which expands to a glob pattern matching all file extensions that tap knows how to load.

The jsx, flow, and ts options are removed. To disable TypeScript and JSX support, remove the @tapjs/typescript plugin with tap plugin rm @tapjs/typescript.

The passes option is added, which will report on all passing tests as well as failures.

The --cov, --no-cov, --coverage and --check-coverage options are removed, along with --lines, --branches, --statements, --functions, and --100. Coverage is always generated, and less than full coverage (or no coverage) is considered a test failure.

Options related to nyc are removed, as nyc is no longer used.

The options --expose-gc, --harmony, --debug, and --debug-brk are removed, as they are extraneous with --node-arg.

The --libtap-settings option is removed, as libtap is no longer used.

The --nyc-version and --parser-version options are removed, in favor of the tap versions command, which prints all versions of relevant modules.

Additional configuration options are added by plugins.

Test Framework Functionality Changes#

The t objects provided to test programs are now proxies that pull in the functionality of all loaded plugins. Mostly everything works the same as before, but inspecting the objects or depending on edge cases might be subtly different.

Much of the added functionality can be turned off by disabling the relevant plugins that provide it, but it is present by default. Similarly, much more functionality can be added by adding plugins, but is not included by default.

The t.intercept, t.capture, and t.captureFn methods are added, enabling "spies" and mocks for methods and properties.

The t.mock method is now t.mockRequire. An asynchronous t.mockImport is added for mocking modules loaded via import(). The t.createMock method can be used to swap out one or more properties of an object passed to t.mockRequire or t.mockImport.

The t.worker method can be used to spawn subtest processes as worker threads. t.isMainThread can be inspected to know if the test is running in the main thread rather than a worker. t.workerData will be set to any data provided to the worker thread.

The t.matchOnly, t.matchStrict, and t.matchOnlyStrict methods are added, along with their negated counterparts.

The t.error, t.doesNotThrow, t.resolves, and t.resolveMatch now show both the origin callsite of the error on failure, as well as the callsite where the test assertion was made.

The expectFail assertion option is removed.

Assertion aliases are moved to the optional @tapjs/synonyms plugin.

The t.testdir() method now stores its fixtures in .tap/fixtures in the project root, rather than tap-testdir-{name} as a sibling to the test file.

t.teardown() and t.after() are the same now.

Multiple methods provided to t.teardown() and t.after() will be run in reverse order.

16.2 - 2022-05-04#

16.1 - 2022-04-24#

16.0 - 2022-03-05#

15.2 - 2022-03-04#

15.1 - 2021-11-16#

Updated treport to use react 17, new versions of ink and other internal deps, so that tap could work on node v17 and quiet some security advisory nags.

15.0 - 2021-03-30#

This is a major refactor of much of tap's internals, and a lot of new features.

BREAKING CHANGES#

NEW FEATURES and BUG FIXES#

DEPENDENCIES and REFACTORING#

14.10 - 2019-11-20#

14.9 - 2019-10-30#

14.8 - 2019-10-20#

14.7 - 2019-10-14#

14.6 - 2019-08-03#

14.5 - 2019-07-28#

14.4 - 2019-07-02#

14.3 - 2019-06-25#

14.2 - 2019-05-28#

14.1 - 2019-05-20#

14.0 - 2019-05-17#

13.1 - 2019-04-28#

13.0 - 2019-04-25#

Faster, prettier, and more powerful. Major enhancements and quite a few breaking changes. Most tests should continue to work fine, it's worth reading through this changelog if you use previous versions of tap more than casually.

Reporting#

The reporting engine has gotten a massive overhaul.

API Updates#

CLI and Runner Changes#

Configuration#

Low Level Stuff#

12.6 2019-03-06#

12.5 2019-01-29#

Add support for ES Modules in all tap test scripts using esm.

12.4 2019-01-22#

Add support for loading typescript .ts files with ts-node.

12.3 2019-01-22#

Add support for loading .mjs files with the experimental module syntax flag set.

12.2 2019-01-22#

Add --comments to print all t.comment() messages to stderr.

Add TAP_CHILD_ID in the environment of test scripts, so that they can differentiate themselves when spinning up servers and such.

12.1 2018-11-12#

Updates to make tap compatible with running in web browsers using browserify or webpack.

12.0 2018-05-16#

Breaking change to support deep matching and pattern matching of objects in Set collections. (Previously, Set contents would only match if they were equal.)

11.0 2017-11-26#

Significant refactoring and speed improvements.

Add t.skip() and t.todo() methods.

Add t.resolves(promise) to assert that a Promise object (or function that returns a Promise) will resolve.

Add t.resolveMatch(promise, pattern) to assert that a Promise object (or function that returns a Promise) will resolve to a value matching the supplied pattern.

Add support for snapshot testing.

Improved implementation of Mocha-like DSL

BREAKING CHANGES:#

10.7 2017-06-24#

Add support for filtering tests using 'only'

Don't show grep/only skips in the default reporter output.

10.6 2017-06-23#

Add support for filtering tests using regular expressions.

10.5 2017-06-20#

Add support for Maps and Sets in t.match(), t.same(), and t.strictSame().

10.4 2017-06-18#

Add t.rejects() assertion.

10.3 2017-03-01#

10.2 2017-02-18#

Variety of minor cleanup fixes, and a debug mode.

10.1 2017-02-07#

Added support for source maps. Stack traces in your jsx and coffeescript files will now be helpful!

Added the -J option to auto-calculate the number of cores on your system, and run that many parallel jobs.

10.0 2017-01-28#

Full rewrite to support parallel tests. Pass -j4 on the command-line to run 4 test files at once in parallel.

This also refactors a lot of the grimier bits of the codebase, splits the one mega-Test class into a proper OOP hierarchy, and pulls a bunch of reusable stuff out into modules.

Somehow, in the process, it also fixed an odd timing bug with beforeEach functions that returned promises.

It truly is a luxury to have a massive pile of tests when it's time to refactor.

The mocha-like DSL is now much more functional, and documented.

Now supports passng -T or --timeout=0 to the CLI to not impose a timeout on tests.

9.0 2017-01-07#

Buffered subtests!

This adds support for outputting subtests in the buffered format, where the summary test point precedes the indented subtest output, rather than coming afterwards.

This sets the stage for parallel tests, coming in v10. Mostly, it's just an update to tap-parser, and a lot of internal clean-up.

Update nyc to v10, which includes some fixes for covering newer JavaScript language features, and support for implicit function names.

Also: remove a lot of excess noise and repetitive stack traces in yaml diagnostics.

8.0 2016-10-25#

Update tmatch to version 3. This makes regular expressions test against the stringified versions of the test object in all t.match() methods. It's a breaking change because it can cause tests to pass that would have failed previously, or vice versa. However, it is more expected, and strongly recommended.

Handle unfinished promise-awaiting tests when the process exits.

Show yaml diagnostics for the first "missing test" failure when a plan is not met, so that the plan can be more easily debugged. (Diagnostics are still excluded for the additional "missing test" failures that are generated, to reduce unnecessary noise.)

Make coverage MUCH FASTER by turning on nyc caching.

7.1 2016-09-06#

Remove a race condition in how Bail out! messages got printed when setting the "bail on failure" option in child tests. Now, whether it's a child process or just a nested subtest, it'll always print Bail out! at the failure level, then again at the top level, with nothing in between.

Support { diagnostic: false } in the options object for failing tests to suppress yaml diagnostics.

Diagnostics are now shown when a synthetic timeout failure is generated for child test processes that ignore SIGTERM and must be killed with SIGKILL.

7.0 2016-08-27#

Move # Subtest commands to the parent level rather than the child level, more like Perl's Test2 family of modules. This is more readable for humans.

Update to version 2 of the tap parser. (This adds support for putting the # Subtest commands at the parent level.)

Support use of a --save and --bail together. Any test files that were skipped due to a bailout are considered "not yet passed", and so get put in the save file.

Forcibly kill any spawned child process tests when the root test exits the parent process, preventing zombie test processes.

Handle SIGTERM signals sent to the main process after the root test has ended. This provides more useful output in the cases where the root test object has explicitly ended or satisfied its plan, but a timeout still occurs because of pending event loop activity.

Prevent for..in loops from iterating inherited keys in many cases, providing resilience against Object.prototype mutations.

Add the --100 flag to set statements, functions, lines, and branches to 100% coverage required.

6.3 2016-07-30#

Let t.doesNotThrow take a string as the first argument.

Bump nyc up to version 7.

The tap lib/ folder is excluded from all stack traces.

6.2 2016-07-15#

Add the --test-arg=<argument> option.

6.1 2016-07-01#

Add support for {diagnostic: true} in test and assert options, to force a YAML diagnostic block after passing child tests and assertions.

6.0 2016-06-30#

Only produce output on stdout if the root TAP test object is interacted with in some way. Simply doing require('tap') no longer prints out the minimum TAP output, which means that you can interact with, for example, var Test = require('tap').Test without causing output side effects.

Add ~/.taprc yaml config file support.

Add the --dump-config command line flag to print out the config options.

Document environment variables used.

Built-in CodeCov.io support has been removed. If you were relying on this, you can add codecov as a devDependency, and then add "posttest": "tap --coverage-report=lcov | codecov" to the scripts section in your package.json file.

5.8 2016-06-24#

Make coverage piping errors non-fatal.

Clean up argument ordering logic in t.throws(). This now works for almost any ordering of arguments, which is unfortunately necessary for historical reasons. Additionally, you can now pass in an Error class to verify the type, which would previously not work properly in some cases.

5.7 2016-02-22#

Report timeout errors in child test scripts much more diligently.

On Unix systems, the child process handles SIGTERM signals by assuming that things are taking too long, dumping a report of all active handles and requests in process, and exiting in error.

On Windows systems (where SIGTERM is always uncatchably fatal), or if a Unix child test process doesn't exit within 1 second (causing a fatal SIGKILL to be sent), the parent generates more comprehensive output to indicate that the child test exited due to a timeout.

5.6 2016-02-17#

Update tmatch to version 2. You can now test objects by supplying their constructor, so t.match(x, { foo: Function, name: String }) would verify that the object has a name string and a foo method.

5.5 2016-02-15#

Add the t.assertAt and t.assertStack properties, to override where an assertion was effectively called from.

5.4 2016-01-31#

Support passing in a class to t.throws, rather than an Error sample object.

5.3 2016-01-31#

Return a Promise object from t.test(), t.spawn(), and t.stdin().

5.2 2016-01-26#

Adds t.beforeEach() and t.afterEach().

5.1 2016-01-16#

All about the cli flags!

Support --node-arg=... and --nyc-arg=... command line flags.

Add support for coverage checking using --statements=95 etc.

Test for executable-ness more consistently across platforms.

5.0 2016-01-03#

Make it impossible to try/catch out of plan/end abuses. Calling t.end() more than once, or having a number of tests that doesn't match the plan() number, is always a failure.

Push thrown errors to the front of the action queue. This means that, even if other things are pending, an uncaught exception or a plan/end bug, will always take the highest priority in what gets output.

Many updates to nyc, spawn-wrap, and foreground-child, so that tap now reliably works on Windows (and a ci to prove it.)

Moved into the tapjs org.

4.0 2015-12-30#

Raise an error if t.end() is explicitly called more than once. This is a breaking change, because it can cause previously-passing tests to fail, if they did t.end() in multiple places.

Support promises returned by mochalike functions.

3.1 2015-12-29#

Support sending coverage output to both codecov.io and coveralls.io.

3.0 2015-12-29#

Upgrade to nyc 5. This means that config.nyc.exclude arrays in package.json now take globs instead of regular expressions.

2.3 2015-11-18#

Use the name of the function supplied to t.test(fn) as the test name if a string name is not provided.

Better support for sparse arrays.

2.2 2015-10-23#

Add support for Codecov.io as well as Coveralls.io.

Catch failures that come after an otherwise successful test set.

Fix timing of t.on('end') so that event fires before the next child test is started, instead of immediately after it.

t.throws() can now be supplied a regexp for the expected Error message.

2.1 2015-10-06#

Exit in failure on root test bailout.

Support promises returned by t.test(fn) function.

2.0 2015-09-27#

Update matching behavior using tmatch. This is a breaking change to t.match, t.similar, t.has, etc., but brings them more in line with what people epirically seem to expect these functions to do.

Deal with pending handles left open when a child process gets a SIGTERM on timeout.

Remove domains in favor of more reliable and less invasive state and error-catching bookkeeping.

1.4 2015-09-02#

Add t.contains() alias for t.match().

Use deeper for deep object similarity testing.

Treat unfinished tests as failures.

Add support for pragmas in TAP output.

1.3 2015-06-23#

Bind all Test methods to object.

Add t.tearDown(), t.autoend(), so that the root export is Just Another Test Object, which just happens to be piping to stdout.

Support getting an error object in bailout()

1.2 2015-05-26#

Better support for exit status codes.

1.1 2015-05-20#

Add coverage using nyc.

If a COVERALLS_REPO_TOKEN is provided, then run tests with coverage, and pipe to coveralls.

1.0 2015-05-06#

Complete rewrite from 0.x.

Child tests implemented as nested TAP output, similar to Perl's Test::More.

0.x#

The 0.x versions used a "flattened" approach to child tests, which requires some bookkeeping.

It worked, mostly, but its primary success was inspiring tape and tap v1 and beyond.