Tests that could not fail

Problem

Two tests in this build passed whatever the code did. A contrast test looped over a list of large-text colour pairs, and the list was empty, so the loop never ran and nothing was checked. A second test searched the built site for a sentence about repository access, and the site had never contained that sentence, so the search could only come back empty. Both showed green, and both made the suite look more thorough than it was.

Symptoms

• The suite was green, yet changing the colours or the page text could not turn either test red.
• A review noticed both by asking what change each test would catch, and finding none.

What Didn't Work

• Trusting the plan. Both tests were written into the plan, so they looked approved. A plan can describe a test that cannot fail.
• Reading the test's name. Each name described a real guard, and the name was all that was real about it.

Solution

Both tests were removed, along with the empty list and the plan lines that described them. Every colour pair already met the stricter contrast rule in the main list, so the large-text list had nothing left to guard. The second test guarded a sentence the site never had, so nothing real was lost.

A test that loops should also check that it looked at something:

const pages = builtPages()
expect(pages.length).toBeGreaterThan(0)
for (const page of pages) {
  // the real assertions
}

Why This Works

A test earns trust only by having failed once for the right reason. A loop over nothing and a search for text that was never there have the same flaw. Neither can reach a state where the assertion is false, so neither says anything about the code.

Prevention

• Write the test, then change the code or the data so the thing under test is wrong, and watch the test fail before trusting it. Put the code back and watch it pass.
• When a test loops or scans, assert that it found at least one item to look at.
• When a test looks for wording, first check that the wording exists in the thing it searches.
• Treat a test that cannot fail as a defect, even when the plan asks for it.

Related Issues

• A CV check that passed cut and added text is the same habit applied to a check that was real but too loose.
• The history check passed with its safeguards removed found a safeguard no test covered.