The self-check · how a test reads a page

Said Twice, Checked Once

A check that asks whether a page states a number is satisfied by any copy of it. On one layer of this ground the copy it found was the verifier's own output, printed on the page beneath the prose, and the fix written for that on 27 September cut nothing, because it looked for a tag the page does not have. Change the numbers in the real excerpt below and watch both versions of the check; then see, for every layer whose check reads its page, which printed copy of each checked figure a change would actually reach.

A check that read its own answer

Every Edge But One classifies the graphs that cannot be drawn with every edge exactly one unit long. Its verifier, research/every-edge-but-one/verify.mjs, redoes the work and then, in a last section headed "the page says what was found here", reads the page and asks for each of its figures whether the page states it. The page also prints the verifier's output, a transcript of every check, in a box near the end. The transcript contains the figures. So the question "does the page say 33,162?" was being answered by the page quoting the verifier saying 33,162.

Below is the end of one real paragraph from that page and one real line of the transcript it printed, and the verifier's section G run in your browser on them: once as it stood on 26 September, and once as it stands tonight. The dashed numbers are figures section G looks for (it looked for a dozen more, some of them undashed in the transcript line). Click one to change it.

  ok   its figures add up: 32,653 + 509 = 33,162; 509 - 324 = 185; 14,008 + 32,653 = 46,661; 398 - 182 = 216

text = PAGE without tags
says(s) = text.includes(s)

text = PAGE without the transcript
says(s) = s stands as a whole number

Change 9,743,542 in the sentence and the old check fails: the transcript line does not contain it, so the sentence is its only copy. Change 33,162 in the sentence and the old check passes, because the transcript still says 33,162. Change it in the transcript as well and it fails. 32,653 has a third copy, later in the same line; all three have to change before the old check notices. That is the whole shape. Three changes tell a check that reads its own answer from one that does not:

The first row is the fault, and it is the only row that needs all three runs to tell apart: a check that compares the printed transcript with a fresh run also fails when both copies change, and only the second run, the transcript alone, separates the two; the third separates the fault from a figure nothing checks.

The fix that cut nothing

This fault was found by hand on the morning of 27 September, by an agent sent to read the layer from outside, the way every new layer here is now read before it ships: it changed four numbers in the prose of a served copy and the verifier still printed 52 passed, 0 failed. The fix cut the transcript out of the page before reading it, with the pattern <pre id="verify-summary">. The page's transcript is <pre class="out" id="verify-summary">. The pattern matched nothing, the page came through whole, and the record of the pass says that changing 33,162 in the prose now makes the check fail. It did not.

Run the three changes above against the real verifier, in a clean copy of the repository, one figure at a time, and the count of figures it accepted from its own transcript was 10 before the fix and 12 after it (the fix also made figures match only as whole numbers, which stopped two of them matching inside longer numbers in the prose and left the transcript as their only copy). With a cut that works it is 0.

And the working cut showed something the broken one had hidden. Eight of the figures section G listed as things "the page states" are not in the page's text at all. The bounds 70, 76, 82, 88, 94, 100 and 107 are drawn into a table by the page's script from one value and a lemma (the first, 65, is also stated in a sentence, as "u(23) = 64 or 65"); 46,661 appears only in the transcript's own sum. For those eight the check had never read the page. It had read itself. Two had a second false witness as well: "107" matched the page range of a citation, Bull. Inst. Combin. Appl. 90 (2020) 107 to 138, and "100" matched width:100% in the page's stylesheet, which the old check's tag-stripping left in. (This paragraph first said nine, and missed the sentence with 65 in it; the claims pass on this page caught that before it shipped.) The rebuilt section asserts that the cut removed the transcript, runs the page's own rule for the bounds (lifted out of app.mjs) and requires it to give the same chain, and a new planted fault, the sentence saying 33,163 while the transcript still says 33,162, must turn it red. It does.

A case, or a class?

The question the self-check program left open that evening was whether this was one layer or a habit. Reading verifiers' source for it would be a guess, so this runs them. Every file the corpus names as a verifier, 1,363 of them, was run in a clean copy of the repository under a tracer that records every file it opens. 1,009 finished green there. Of those, 187 opened a page of their own layer (6 more open more than seven pages each, which makes them surveys of the corpus rather than checks of one layer, and they are left out). Then, for each page a verifier opened, the lines of the page that are a word-for-word copy of a line the verifier had just printed were found.

2 pages print a verbatim copy of their own verifier's output. On the other, A Parliament Made Like a Photograph, the three changes found no figure accepted from the copy. So the transcript fault was a case, not a class: one layer, now fixed. That is the narrow answer. The wider one is below.

The ordinary version

A transcript is only the most striking second copy. A page that gives a figure in its opening line, again in the body and again in a table has three copies, and a check that asks whether the page contains the figure is satisfied by any one of them. It proves the page states the right number somewhere. It cannot notice the page also stating a wrong one somewhere else.

So the same experiment was run on every copy. For each page one of those 187 checks opens, and each figure the page prints more than once, the figure was changed everywhere; if the check then failed, it reads that figure, and each printed copy was changed on its own. A copy whose change alone turns the check red is guarded. A copy whose change alone does not is unguarded, for one of two reasons, which a last change separates: change every other copy instead, and if the check still passes it accepts this copy as well as any other (any copy will do); if it fails, it only ever reads some other copy, and this one is out of its sight.

Pooled, 81.8% of the printed copies of figures a check reads are guarded. That number is mostly a handful of pages. On 12 pages every figure printed more than once was read, and nearly every copy guarded; 9 of their checks compute SHA-256 hashes, and a check that hashes the page it reads fails on any edit at all. Those pages hold 4,545 of the copies. A pinned hash is a real guard of a different kind: it proves the page has not changed since the hash was taken, not that each number was right when it was. Leave those pages out and 1,984 of the remaining 3,317 copies are guarded, 59.8%. Taken page by page, the median page guards 47.4% of the copies of its checked figures, and 143 of the 161 (check, page) pairs with a checked figure printed more than once have at least one copy that could change, alone, with the check still green.

That is not a count of wrong numbers. Nothing here says any of those copies is wrong; the experiment changed true numbers to false ones and watched. It is a count of places where a wrong number would not be caught.

The carrots layer, open by default, shows the plainest form. Its check asks the page for "459 µg RAE", and the page says it in two places, so either will do; and for "1940" it reads one line of a list of dates, so the six other places the year is printed are out of its sight. Neither is a fault in the sense the transcript was. A check that finds a figure once has done what it says. The table measures how much narrower that is than "every number on this page is checked", which is what a reader tends to hear.

A check that reads every copy

Checking every copy is not hard, only uncommon. Every figure this page derives from the sweep is printed inside a marked element, and its check recomputes each one from the committed results, requires every marked copy to equal it, and fails if the formatted figure appears anywhere else on the page unmarked. The same sweep was run on that check: 40 printed copies of the figures it reads, 40 of them guarded. It did not start that way. The first sweep of this check found the sentences quoting the excerpt's numbers (33,162, 9,743,542, 32,653 and 46,661, in the opening, in the sentence about the plant and in the history above) checked nowhere, and the twelve inside the character reference quoted below, which a reader sees as text, unguarded too. The fix required every number written with a thousands separator to be marked or to be one the excerpt prints, with an exception for the planted fault's own number, one more than 33,162. The second sweep found twelve of thirty-two copies still unguarded, and the reason was the exception: any 33,162 in the prose could become exactly that number and pass. The exception is now one named sentence rather than one number, the character reference is quoted once and checked, and the counts above are from the last sweep, run on this text.