Node TAP 21.0.1

Interoperability with the Node.js Built-in Test Runner

tl;dr - Use the tap runner to run tests written with node:test or run tap tests with node --test and it Just Works with full interoperability. Mix and match how you see fit.

The best way to appreciate how these two things work together is with an example. The tapjs/node-test-example repo is a module with a shocking number of bugs for how little code it is, with a test written using node:test and effectively the same test written using tap.

When you run npm test, it'll run with both tap and node --test.

git clone git@github.com:tapjs/node-test-example.git
cd node-test-example
npm install
npm test # run both with tap, and both with node --test

The test:mix and test:cross show using the node --test and tap runners so that they dump coverage into the same folder. Then you can use tap report to report on it.

In all cases, you can see that the results are pretty similar.

Differences#

Of course, the two runners produce very different output overall, but they should both be pretty sensible.

Personally, I think the tap runner is a lot more useful, and certainly if you write tests in TypeScript (or use tap's import mocking) it's nice to not have to specify the --loader and --import arguments explicitly.

But on the flip side, that fanciness comes with a cost. With TypeScript disabled, tap runs these two tests in about 450ms on my system (350ms or so with coverage disabled), while node --test does it in around 170ms. In both cases, the test/tap.test.js test takes around 150ms to run, and the test/node.test.js takes under 10ms.

Real world tests doing complicated stuff would show a less dramatic difference, so this is in no way a representative benchmark, but as always, performance and features are fundamentally opposed, because features require running code, and not running code is always faster.

You Do You#

The goal of the node:test interoperability in node-tap is to make it possible for you to get the best of both worlds. You could have part of your test suite written as node:test tests, if they don't need t.mockImport or TypeScript, and other tests written in tap that are just more convenient to run with a runner that knows which loaders to apply.

Enough talk! Show the output!#

Running with tap:

 FAIL  test/node.test.js 2 failed of 4 6.834ms
    suite of tests that fail > uhoh, this one throws
    suite of tests that fail > failer
 FAIL  test/tap.test.js 3 failed of 18 340ms
    suite of tests that fail > uhoh, this one throws > Invalid time value lib/index.mjs:11:43
    suite of tests that fail > failer > should be equal test/tap.test.js:35:7
    suite of tests that fail > failer > should be equal test/tap.test.js:37:7
                                    
                                            
  🌈   TEST COMPLETE 🌈                                                                         
                                                                            
                                                                             
 FAIL  test/node.test.js 2 failed of 4 6.834ms
    suite of tests that fail > uhoh, this one throws
    test/node.test.js                                                       
    20 })                                                                   
    21                                                                      
    22 test('suite of tests that fail', async t => {                        
    23   await t.test('uhoh, this one throws', () => {                      
    ━━━━━━━━━━━━━                                                          
    24     assert.equal(thrower(0), '1970-01-01T00:00:00.000Z')             
    25     assert.equal(thrower(1234567891011), '2009-02-13T23:31:31.011Z') 
    26     assert.equal(thrower({}), 'Invalid Date')                        
    27   })                                                                 
    error origin: lib/index.mjs                                  
     8                                                           
     9 // This is a function that throws, to show how both       
    10 // handle errors.                                         
    11 export const thrower = (n) => new Date(n).toISOString()   
    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━               
    12                                                           
    13 // one that fails, to show how failures are handled       
    14 export const failer = (n) => String(n + 1)                
    error: Invalid time value
    code: ERR_TEST_FAILURE
    failureType: testCodeFailure
    name: RangeError
    Date.toISOString (<anonymous>)
    thrower (lib/index.mjs:11:43)
    TestContext.<anonymous> (test/node.test.js:26:18)
    TestContext.<anonymous> (test/node.test.js:23:11)

    suite of tests that fail > failer
    test/node.test.js                                                       
    26     assert.equal(thrower({}), 'Invalid Date')                        
    27   })                                                                 
    28                                                                      
    29   await t.test('failer', () => {                                     
    ━━━━━━━━━━━━━                                                          
    30     assert.equal(failer(1), '2')                                     
    31     assert.equal(failer(-1), '0')                                    
    32     // expect to convert string numbers to Number, but doesn't       
    33     assert.equal(failer('1'), '2')                                   
    error origin: test/node.test.js                                         
    30     assert.equal(failer(1), '2')                                     
    31     assert.equal(failer(-1), '0')                                    
    32     // expect to convert string numbers to Number, but doesn't       
    33     assert.equal(failer('1'), '2')                                   
    ━━━━━━━━━━━━━━                                                         
    34     // expect to convert non-numerics to 0, but it doesn't           
    35     assert.equal(failer({}), '1')                                    
    36   })                                                                 
    37 })                                                                   
    --- expected   
    +++ actual     
    @@ -1,1 +1,1 @@
    -"2"           
    +"11"          
    error: "'11' == '2'"
    code: ERR_ASSERTION
    failureType: testCodeFailure
    name: AssertionError
    operator: ==
    TestContext.<anonymous> (test/node.test.js:33:12)
    TestContext.<anonymous> (test/node.test.js:29:11)

 FAIL  test/tap.test.js 3 failed of 18 340ms
    suite of tests that fail > uhoh, this one throws > Invalid time value
    lib/index.mjs                                                
     8                                                           
     9 // This is a function that throws, to show how both       
    10 // handle errors.                                         
    11 export const thrower = (n) => new Date(n).toISOString()   
    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━               
    12                                                           
    13 // one that fails, to show how failures are handled       
    14 export const failer = (n) => String(n + 1)                
    type: RangeError
    tapCaught: testFunctionThrow
    Date.toISOString (<anonymous>)
    thrower (lib/index.mjs:11:43)
    Test.<anonymous> (test/tap.test.js:27:13)

    suite of tests that fail > failer > should be equal
    test/tap.test.js                                                        
    32     t.equal(failer(1), '2')                                          
    33     t.equal(failer(-1), '0')                                         
    34     // expect to convert string numbers to Number, but doesn't       
    35     t.equal(failer('1'), '2')                                        
    ━━━━━━━━━                                                              
    36     // expect to convert non-numerics to 0, but it doesn't           
    37     t.equal(failer({}), '1')                                         
    38     t.end()                                                          
    39   })                                                                 
    --- expected   
    +++ actual     
    @@ -1,1 +1,1 @@
    -2             
    +11            
    compare: ===
    Test.<anonymous> (test/tap.test.js:35:7)
    Test.<anonymous> (test/tap.test.js:31:5)
    test/tap.test.js:23:3

    suite of tests that fail > failer > should be equal
    test/tap.test.js                                                        
    34     // expect to convert string numbers to Number, but doesn't       
    35     t.equal(failer('1'), '2')                                        
    36     // expect to convert non-numerics to 0, but it doesn't           
    37     t.equal(failer({}), '1')                                         
    ━━━━━━━━━                                                              
    38     t.end()                                                          
    39   })                                                                 
    40                                                                      
    41   t.end()                                                            
    --- expected     
    +++ actual       
    @@ -1,1 +1,1 @@  
    -1               
    +[object Object]1
    compare: ===
    Test.<anonymous> (test/tap.test.js:37:7)
    Test.<anonymous> (test/tap.test.js:31:5)
    test/tap.test.js:23:3

Asserts:  17 pass  5 fail  22 of 22 complete
Suites:    0 pass  2 fail    2 of 2 complete

# { total: 22, pass: 17, fail: 5 }
# time=459.924ms

Running with node --test:

$ node --test
   add (0.569917ms)
   stringOrNull (0.063833ms)
   suite of tests that fail
     uhoh, this one throws (0.910959ms)
    RangeError [Error]: Invalid time value
        at Date.toISOString (<anonymous>)
        at thrower (file:///Users/isaacs/dev/tapjs/node-test-example/lib/index.mjs:11:43)
        at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:26:18)
        at Test.runInAsyncScope (node:async_hooks:206:9)
        at Test.run (node:internal/test_runner/test:631:25)
        at Test.start (node:internal/test_runner/test:542:17)
        at TestContext.test (node:internal/test_runner/test:167:20)
        at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:23:11)
        at Test.runInAsyncScope (node:async_hooks:206:9)
        at Test.run (node:internal/test_runner/test:631:25)

     failer (0.532708ms)
    AssertionError [ERR_ASSERTION]: '11' == '2'
        at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:33:12)
        at Test.runInAsyncScope (node:async_hooks:206:9)
        at Test.run (node:internal/test_runner/test:631:25)
        at Test.start (node:internal/test_runner/test:542:17)
        at TestContext.test (node:internal/test_runner/test:167:20)
        at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:29:11)
        at async Test.run (node:internal/test_runner/test:632:9)
        at async Test.processPendingSubtests (node:internal/test_runner/test:374:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: '11',
      expected: '2',
      operator: '=='
    }

   suite of tests that fail (1.684292ms)

   add (1.774ms)
   stringOrNull (1.091ms)
   suite of tests that fail
     uhoh, this one throws (10.016ms)
    Error: Invalid time value
    | // This is a function that throws, to show how both
    | // handle errors.
    | export const thrower = (n) => new Date(n).toISOString()
    | ------------------------------------------^
    | 
    | // one that fails, to show how failures are handled
        at Date.toISOString (<anonymous>)
        at thrower (/Users/isaacs/dev/tapjs/node-test-example/lib/index.mjs:11:43)
        at Test.<anonymous> (/Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:27:13) {
      type: 'RangeError',
      tapCaught: 'testFunctionThrow'
    }

     failer (3.676ms)
    Error: should be equal
    --- expected                                                               
    +++ actual                                                                 
    @@ -1,1 +1,1 @@                                                            
    -2                                                                         
    +11                                                                        
    |     t.equal(failer(-1), '0')
    |     // expect to convert string numbers to Number, but doesn't
    |     t.equal(failer('1'), '2')
    | ------^
    |     // expect to convert non-numerics to 0, but it doesn't
    |     t.equal(failer({}), '1')
        at Test.<anonymous> (/Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:35:7)
        at Test.<anonymous> (/Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:31:5)
        at /Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:23:3 {
      compare: '==='
    }

   suite of tests that fail (17.681ms)

   tests 9
   suites 1
   pass 4
   fail 5
   cancelled 0
   skipped 0
   todo 0
   duration_ms 160.809375

   failing tests:

test at file:/Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:23:11
   uhoh, this one throws (0.910959ms)
  RangeError [Error]: Invalid time value
      at Date.toISOString (<anonymous>)
      at thrower (file:///Users/isaacs/dev/tapjs/node-test-example/lib/index.mjs:11:43)
      at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:26:18)
      at Test.runInAsyncScope (node:async_hooks:206:9)
      at Test.run (node:internal/test_runner/test:631:25)
      at Test.start (node:internal/test_runner/test:542:17)
      at TestContext.test (node:internal/test_runner/test:167:20)
      at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:23:11)
      at Test.runInAsyncScope (node:async_hooks:206:9)
      at Test.run (node:internal/test_runner/test:631:25)

test at file:/Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:29:11
   failer (0.532708ms)
  AssertionError [ERR_ASSERTION]: '11' == '2'
      at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:33:12)
      at Test.runInAsyncScope (node:async_hooks:206:9)
      at Test.run (node:internal/test_runner/test:631:25)
      at Test.start (node:internal/test_runner/test:542:17)
      at TestContext.test (node:internal/test_runner/test:167:20)
      at TestContext.<anonymous> (file:///Users/isaacs/dev/tapjs/node-test-example/test/node.test.js:29:11)
      at async Test.run (node:internal/test_runner/test:632:9)
      at async Test.processPendingSubtests (node:internal/test_runner/test:374:7) {
    generatedMessage: true,
    code: 'ERR_ASSERTION',
    actual: '11',
    expected: '2',
    operator: '=='
  }

test at test/tap.test.js:24:5
   uhoh, this one throws (10.016ms)
  Error: Invalid time value
  | // This is a function that throws, to show how both
  | // handle errors.
  | export const thrower = (n) => new Date(n).toISOString()
  | ------------------------------------------^
  | 
  | // one that fails, to show how failures are handled
      at Date.toISOString (<anonymous>)
      at thrower (/Users/isaacs/dev/tapjs/node-test-example/lib/index.mjs:11:43)
      at Test.<anonymous> (/Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:27:13) {
    type: 'RangeError',
    tapCaught: 'testFunctionThrow'
  }

test at test/tap.test.js:31:5
   failer (3.676ms)
  Error: should be equal
  --- expected                                                               
  +++ actual                                                                 
  @@ -1,1 +1,1 @@                                                            
  -2                                                                         
  +11                                                                        
  |     t.equal(failer(-1), '0')
  |     // expect to convert string numbers to Number, but doesn't
  |     t.equal(failer('1'), '2')
  | ------^
  |     // expect to convert non-numerics to 0, but it doesn't
  |     t.equal(failer({}), '1')
      at Test.<anonymous> (/Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:35:7)
      at Test.<anonymous> (/Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:31:5)
      at /Users/isaacs/dev/tapjs/node-test-example/test/tap.test.js:23:3 {
    compare: '==='
  }