Double Output with Mocha and Assert

If you use Mocha and Node.js assert you may have become used to seeing test failures like this.

1) assert.strictEqual

  0 passing (4ms)
  1 failing

  1) assert.strictEqual:

      AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
+ actual - expected

+ 'foofoo'
- 'barbar'
      ^
      + expected - actual

      -foofoo
      +barbar

      at Context.<anonymous> (index.js:4:10)
      at processImmediate (internal/timers.js:439:21)

In fact, you’re probably so used to seeing this that you don’t even notice the three things that are wrong with it.

1. The error message/diff is displayed twice
2. One of the messages is weirdly outdented
3. The ordering and color of “actual” and “expected” between the messages are inexplicably both inverted

Now that you’ve seen it you can’t unsee it 🙈 It had been bugging me for a long time and I finally decided to get to the bottom of it. Here’s what I found.

This is the test that generated that output.

it("assert.strictEqual", () => {
  assert.strictEqual("foofoo", "barbar")
})

When exploring a problem I try to change small things one at a time to see what happens. So let’s change assert.strictEqual to assert.equal.

it("assert.equal", () => {
  assert.equal("foofoo", "barbar")})

1) assert.equal

  0 passing (4ms)
  1 failing

  1) assert.equal:

      AssertionError [ERR_ASSERTION]: 'foofoo' !== 'barbar'
      + expected - actual

      -foofoo
      +barbar

      at Context.<anonymous> (index.js:4:10)
      at processImmediate (internal/timers.js:439:21)

That output looks more like what you’d expect. Concise and indented correctly. But I would not have expected that to make a difference. What’s going on here?

To figure it out, let’s catch the errors we’re creating with the assertions and introspect the error objects themselves.

it("assert.strictEqual", () => {
  try {    assert.equal("foofoo", "barbar")
  } catch (error) {    console.log(error)  }})

it("assert.strictEqual with shorter strings", () => {
  try {    assert.strictEqual("foo", "bar")
  } catch (error) {    console.log(error)  }})

This is the log from assert.strictEqual.

AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
+ actual - expected

+ 'foofoo'
- 'barbar'

{
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: 'foofoo',
  expected: 'barbar',
  operator: 'strictEqual'
}

And this is the log from assert.equal.

AssertionError [ERR_ASSERTION]: 'foofoo' == 'barbar'

{
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: 'foofoo',
  expected: 'barbar',
  operator: '=='
}

I’ve massaged the output a little bit to make it easier to understand what’s going on.

First you can see an error “message”. Then you can see the properties of the Error class. The properties are used by the Error class to generate the error message.

If we go back to our Mocha output and subtract the error messages we’re left with this.

1) assert.strictEqual

  0 passing (4ms)
  1 failing

  1) assert.strictEqual:

      + expected - actual

      -foofoo
      +barbar

      at Context.<anonymous> (index.js:4:10)
      at processImmediate (internal/timers.js:439:21)

1) assert.equal

  0 passing (4ms)
  1 failing

  1) assert.equal:

      + expected - actual

      -foofoo
      +barbar

      at Context.<anonymous> (index.js:4:10)
      at processImmediate (internal/timers.js:439:21)

Now the output is effectively identical. I think I’m finally starting to see the shape of what happened here.

The remaining output must be generated by Mocha. In the same way that the Error class uses its properties to generate a useful error message, Mocha also uses those properties to make a helpful “diff” of the actual and expected values.

This helps us partially answer our question. The reason why the output is “doubled” when using assert.strictEqual is because the output is coming from two different places: some from assert and some from Mocha.

But then the question becomes “How did we end up here?” because showing two diffs makes absolutely no sense.

assert.equal and assert.strictEqual have both been part of Node from the very beginning, so you’d think their output would be identical—they’re just different ways of asserting equality. Let’s go spelunking in the Node codebase to see what’s going on.

The error message gets generated in the AssertionError class. Here’s the relevant spot.

if (operator === 'deepStrictEqual' || operator === 'strictEqual') {
  super(createErrDiff(actual, expected, operator));

assert.strictEqual is being handled differently from assert.equal when the error message is generated and gets a special diff. Has it always been that way?

If you spin your way back through the commit history you’ll eventually find this pull request from late 2017 with this abridged description.

The current assert errors are actually pretty bad when it comes to objects. This implements a simple way to add diffs as default but only in assert strict mode.

…

Activating the diff always would mean we have to change all our assert tests and I did not want to do that, besides the fact that some people might partially rely on the message output.

So basically what happened here is Node used to have mediocre error messages and so test frameworks like Mocha generated additional information to make debugging easier.

Then a few years ago someone improved Node’s error messages but only for strict assertions. I think that’s pretty confusing but in fairness to them, non-strict assertions are deprecated and you’re not supposed to use them.

So that’s it. The reason why Mocha’s test output looks like garbage with assert is because about 8 years after Node and Mocha were created the implementation of assert changed.

That’s unfortunate. Now that we know the reason, what can we do about it?

The solution is painfully simple but I think it doesn’t become clear what it is until you understand what changed and why.

Mocha’s diffing is configurable. It defaults to on but we need to turn it off and let assert handle the diffing.

Run the mocha command with --diff false or set it in your config.

// .mocharc.js

module.exports = {
  diff: false,
}

This is what the output looks like now.

1) assert.strictEqual

  0 passing (4ms)
  1 failing

  1) assert.strictEqual:
      AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
+ actual - expected

+ 'foofoo'
- 'barbar'
      at Context.<anonymous> (index.js:4:10)
      at processImmediate (internal/timers.js:439:21)

That’s definitely an improvement, although the error message is still outdented grossly. Unfortunately fixing the indentation has no easy, direct solution. Ideally this would be fixed in Mocha, but you’d have to fix every single Mocha reporter and that could necessitate some breaking or undesirable changes to the output.

The only alternative is to use a different assertion library. The biggest improvement for the least amount of effort is probably Power Assert.

Power Assert has an interface that looks like assert but with truly next-level error messages. Here are two examples from the docs.

assert(ary.indexOf(zero) === two)
        |   |       |     |   |
        |   |       |     |   2
        |   -1      0     false
        [1,2,3]

assert(types[index].name === bob.name)
        |    ||      |    |   |   |
        |    ||      |    |   |   "bob"
        |    ||      |    |   Person{name:"bob",age:5}
        |    ||      |    false
        |    |11     "alice"
        |    Person{name:"alice",age:3}
        ["string",98.6,true,false,null,undefined,#Array#,#Object#,NaN,Infinity,/^not/,#Person#]

Yeah 🤯 I didn’t even know how badly I needed this until I saw it for the first time.

The way Power Assert is implemented it uses transpilation to enhance assert, so you can use it without having to change your test suite at all. You can just install it and immediately start getting better error messages!