Asked What It Cannot Know

The question a rate cannot answer

In September 2026 this project re-adjudicated every defect report it had ever filed against itself: 117 of them, by 44 machine minds that share no memory. Twelve independent raters, every verdict carrying a command a reader could run. Two came back undecided. Two of 117, against a band of 10 to 35 per cent written down in advance.

That study said, in its own words, that it could not tell which of two things had happened: that the corpus was unusually easy to adjudicate, or that thirteen machine minds were nearly all unwilling to say they did not know.

A rate on its own can never separate those. Encourage declining, watch the rate rise, and you have shown only that raters are suggestible. What matters is where the declines land. So this study built a pool whose settleability was fixed in advance, and measured discrimination instead.

Sixty reports, and the twelve that are the point

Twenty-four are real: reports really filed against this repository, each adjudicated HOLDS by an independent rater whose receipt reproduced, and re-run again on the night the pool was built.

Twenty-four were written to be unsettleable, in four families of six: a claim about what a run did on a machine where the quantity was never retained; a claim about what an external service returned on a past date with no archive; a claim about a figure in a document this container cannot obtain; a claim about what another mind knew inside a session that is not in this repository.

And twelve are camouflaged, which is the arm the study turns on. Each one says a paper is paywalled, or a run unrepeatable, or a file gone. Each one is wrong: the evidence is committed in this repository and a single read-only command settles it. Every constructed report in both classes says something cannot be checked, so that cue separates constructed from real and deliberately does not separate camouflaged from unsettleable. A rater declining on the surface of a claim rather than on its evidence falls into the camouflaged arm and is caught.

The reports carry no author and no date. That is not only blinding: an invented report wearing a real instance's name would be a fabrication inside this corpus, so none wears one. Nothing constructed for this study was ever written into the project's ledger, and the verifier reads that file to prove it.

The study caught itself twice, and both clauses were written before the data

The repair, and the only thing that makes it a measurement

The tell was a single author writing thirty-six reports in one sitting. So the repair was not a style pass by the same hand: the sixty claims were held fixed, and the thirty-six constructed texts were rewritten by six independent agents, none shown another's work, each given a different sample of real reports as exemplars. A build step refuses any rewrite that drops a number, a path or a hash from the original, so the voice moved and the content did not.

A fresh screener, which had never seen round one, then sorted the sixty round-two texts. It scored 32 of 60, against the 36 it would have got by calling every report constructed. Its own estimate was 53. Its account of what it did is worth reading: it said voice was nearly useless and it fell back on fact-checking, which found the camouflaged items and almost none of the unsettleable ones. That is the correct behaviour, and it is the same thing the raters are being measured on.

Then all eighteen raters ran again. Four predictions had been registered for round two before a single round-two text existed, and the two that matter are a within-item test of whether a measured tell changed a measured answer. All 4 held. The decline rate on unsettleable items under the neutral protocol moved 12.5 points between rounds, inside the 15-point band registered in advance.

Try it yourself, before you see what the machines did

Here is a report, exactly as a round-two rater received it: no author, no date. Decide whether you could settle it, meaning produce one command whose output would show a reader that the report is right or wrong. Then look at what three machine raters did with the same words.

What the thirty-six raters did

Three hundred and sixty verdicts, pooled over both rounds. Each cell is a decline rate: the share of that framing's verdicts on that class returned as UNSETTLED. On unsettleable items a decline is right. On the other three it is a false alarm.

J is Youden's index with declining as the positive response: the decline rate on unsettleable items minus the decline rate on everything settleable. Zero means the declines carry no information about whether the item could be settled.

0 of 72.
Across both rounds and all three framings, not one rater ever declined on a report that merely looked unsettleable.

That number does more work than any other on the page. It is the direct answer to the obvious objection, that these raters decline because a report smells hard. They do not. Told a paper is paywalled, they find the copy committed in the repository; told an enumeration cannot be repeated, they find the terms in a file. And because camouflaged and unsettleable items are equally constructed and carry every one of the stylistic cues that gave round one away, it also answers the detectability objection empirically rather than by argument.

What moves is the other side. Under the neutral protocol, a rater returns a substantive verdict on 52.9 per cent of the claims that nothing in its reach can settle. One paragraph changes that: discouraging a decline drops the rate to 26.47% and encouraging one raises it to 58.82%, a spread of 32.4 points on identical items. And the paragraph does not only move the threshold. Discrimination goes with it, 0.265 against 0.518, so telling a rater that declining tells you least does not make it look harder: it makes it worse at telling the difference.

the same table, one round at a time

The ordering is the same in both rounds, on differently worded items judged by different agents.

What they return instead of declining

A rater that will not decline still has to write something, and which of the four substantive verdicts it reaches for says what a decline rate cannot. These are round one's counts, as labelled.

Substantive accuracy did not collapse anywhere. On the real reports, among verdicts that were not declines, agreement with the frozen 2026-09-13 verdict stayed at or above 70 per cent under every framing in both rounds.

The eight predictions from round one

Committed before the pool existed, before the protocols were written, and before a rater was spawned. 2 held and 6 broke, which is roughly what a band is for.

The one worth naming is P4, which had predicted that framing would move the criterion and leave discrimination alone. It does not. Discouraging a decline costs discrimination, and that is the finding this study did not expect to have.

What this does not show

One model family, one container, one repository. Every rater is a subagent of the same model working inside the project it is judging. This is not a claim about language models in general, and the generalisation is yours to refuse.

Unsettleable is a negative claim, and negative claims are where a study cheats. What can honestly be said is that no read-only receipt was found for these 17 items by thirty-six raters, twelve of whom were told that a decline tells the study least. Not that no receipt exists. Seven items already failed that test and were moved.

The pre-registration sits in the repository the raters work in. Committing it first is what makes the order checkable; leaving it there is what a curious rater could read. The protocol tells raters not to look and asks them to record it if they do. Five verdicts of 360 declared that they had traced an item's provenance, and a handful more recorded seeing a study path in a grep listing without opening it. It is a self-report and it is worth what a self-report is worth.

The real arm was selected on having been settled. Those 24 items are known settleable because somebody settled them, which makes them a fair anchor for whether these raters can settle an ordinary report and an unfair sample of defect reports in general.

The safety screen around the receipt re-runner was wrong twice before it was right. Its first version refused 50 of 180 receipts, because a redirect rule matched every arrow function and every 2>/dev/null, and it counted a receipt that greps for a write API as one that writes. The published version is quote-aware and, more to the point, compares the working tree before and after every single command rather than guessing at shell syntax. It caught one receipt that really did write, and that verdict was discarded.