A CV check that passed cut and added text

Problem

The page's CV text has to be the CV's text, word for word, and the PDF is the judge. The first check took each rendered line and asked whether it was a substring of the PDF's extracted text. That question is far too easy to pass. A bullet cut in half is still a substring. A dropped entry leaves every remaining line a substring. A sentence added to a section passes when each line is only checked on its own.

Symptoms

• Cutting the end off a bullet kept the suite green.
• Removing a whole entry kept the suite green.
• Adding an unmarked sentence inside a section kept the suite green.
• The PDF's extracted text did not follow the page's order. Some job headings came out after later sections, so a plain top to bottom comparison of the whole document did not line up.

What Didn't Work

• A per-line substring check on its own. It only proves each line exists somewhere.
• Comparing whole sections in page order. Because the extracted text puts some headings after later sections, an honest section failed the comparison.

Solution

The check was tightened in three layers, each added after the one before was shown to be beatable by a deliberate edit:

1. Contiguous runs. The lines of a section, joined in page order, must appear as one unbroken run in the PDF text, so a cut or dropped line breaks the run.
2. A length check per section. The squashed length of a section's text must match the length of the same section in the PDF, which catches a cut at the very start or end of a run.
3. A whole-section text comparison. The section's full text, with whitespace squashed, must equal the PDF's text for it, which catches an added sentence that the length check could not tell from a rewording.

Because the PDF lists some headings after Education, Work Experience, Selected Projects and Education are measured as one slice rather than three.

Why This Works

A substring check asks whether the page text is contained in the PDF. The fix asks whether the two are the same. Equal text in the same order, with equal length, leaves no room for cutting, dropping or adding.

Prevention

• When a test compares against a source of truth, check equality, not containment.
• Prove the check with deliberate damage: cut a bullet, drop an entry, add a sentence. Every one of them must fail the check before it is trusted.
• Look at the extracted text before assuming it follows reading order. Extraction from a PDF can reorder blocks.

Related Issues

• Tests that could not fail describes the habit of watching a new test fail first.