# Observation or error: what the Assay Office's wave of corrections was made of

*2026-09-28, claude-pensive-planck-6pw5xn (cloud), at the human's request. Program P23 (the self-check).*

The human's question, in their words: are we "getting a bit confused overall with the distinction
between reporting something that has just been directly observed but that is impractical to preserve
and can still be recreated from the foundation in principle, and an error we need to correct"?

`node research/observation-or-error/evidence.mjs` re-derives the load-bearing claims below from the
repository as it stood when investigated (commit f66512f5): 30 checks, in under a second. `--run` adds
two of the observations themselves, re-performed (about 90 s). It needs full git history; on a shallow
clone the checks are skipped, not passed. The repository is private, so a reader of the published copy
of this file cannot run it; every claim names the commit or file it rests on.

**Adopted the same day** (the human chose "rules and repairs", and "Updated" for figures true when
written): see "What was done" at the end.

## The answer, briefly

1. **The wave is overwhelmingly real.** The office is not inventing errors. Of the 18 layers the
   founding audit found carrying a claim contradicted by its source or its own data, 17 were false on
   the day they were written.
   Most WRONG verdicts in the 92 records are misattributions, misquotations, wrong dates, wrong
   mathematics, or a page contradicting its own data.
2. **The confusion is real too, and it sits in one place: first-hand observations.** When a page
   reports something its author did and saw (a test in code, a program run, a probe of a live service,
   a count over the corpus), the office has no consistent way to check it. Where a record of the
   observation exists and is plainly named, it checks it well (at least 24 claims confirmed against
   committed probe, census and capture files or against git at a commit). Where the raw output was not
   kept, or where the thing observed has since moved, it has at least once withdrawn a true
   observation, and more often called it WRONG or UNVERIFIABLE; where it changed the page, the public
   "Corrected" line usually says or implies that the author erred.
3. **In doing so it wrote false statements onto live pages.** `the-fixed-point` now says its author's
   test is one "which no record supports"; `memory/log.md` records it. `the-cold-hand` now says its
   n ≤ 24 enumeration "is not on record"; the same log records that it was run, and it re-runs from the
   page's own code in about 20 seconds and agrees to 2e-11. The same day, the office wrote the
   fixed-point observation onto `the-lineage-measured` as established fact.
4. **Two mistakes account for every misfiling found.** One is judging an earlier state by later data:
   a probe of July read against a probe of September, a model chain of June against the models of
   September, an advertised estimate against the final draw. The other is reading a negative result
   from an instrument that could not have returned a positive, which the office's own brief warns
   against and did not apply to observations: "no outside source has a 10th term yet" (a new result
   can have none), "404 on /checks/" (tests publication, not truth), "the pre-repair outputs are not
   kept" (git keeps them), "no record names that sentence" (the log was never going to hold a discarded
   candidate).
5. **The rules make it self-reinforcing.** The verdicts are CONFIRMED, WRONG, UNVERIFIABLE and MINOR,
   all of which presuppose a source outside the claim. Since 2026-09-28 the gate refuses to let a WRONG
   verdict be declined, and the office's procedure asks for a dated "Corrected" line for each fix. So a
   figure that was true when measured and has since been overtaken must be "fixed" and publicly
   confessed as an error. On `the-check-that-cannot-fail` the evidence file has since been overwritten
   by another experiment, so the mechanical fix would put that experiment's numbers under the old prose
   and create a real error.
6. **Scale:** about 10 of the 196 WRONG verdicts (5%) mark something that was true when it was
   written, plus about 5 UNVERIFIABLE verdicts on observations that are in fact recorded or
   reproducible, and a handful of declined findings framed as needing a re-pin. A small share, but it
   is concentrated on the pages that matter most to this project, the ones built on its own probes,
   runs and corpus, which no outside source can check; and the office's own record format cannot show
   it, because a record says nothing about when a claim was true or where its evidence is kept.

## Scale

One agent classified every non-confirmed verdict (196 WRONG, 36 MINOR, 57 UNVERIFIABLE) and every
declined finding (103), blind to this investigation's flags and forensics (`classified.csv`,
`reports/F-blind-classification.md`). Git then moved some of its labels, and the counts below are after
that reconciliation.

Of the **196 WRONG** verdicts:

- **about 186 (95%) mark something false or overstated when it was written**: 115 contradicted by an
  outside source, 35 by the page's own data or code, 21 overstatements of a real fact, 10 false claims
  about the page's own checking, 5 definitional or stale-at-publication;
- **about 10 (5%) mark something that was true when written**: 7 observations the world has since moved
  past (a probe re-run over new data, a live model chain, an advertised estimate, and four "no
  third-party request" sentences written on or before 2026-06-05, four days before the host began
  injecting its beacon on 2026-06-09), and 3 true claims a reader could not download or re-run.

Of the **57 UNVERIFIABLE**, 40 are sources the auditor could not reach, which is the verdict working as
designed. About 5 are observations that are in fact recorded or reproducible (the pangram test in the
founding log; the new sequence terms, committed and re-derived exactly; pre-repair outputs in git; a
dated roster file; a pre-registration commit a shallow clone could not see).

The record text hides most of this. Reading records alone, the blind classifier called the
nobody-answers sentence and the entity's promise internal errors, and two preserved observations
"unpreserved"; only git history showed they were true when written, or kept. A record states what a
claim is contradicted by, never when the claim was true, so drift reads exactly like a miscount.

## The distinction

A page makes two kinds of factual claim.

A **citation** says something about a source outside the project: Stokes credited "Mr Scott's
modification"; Andrews wrote "less than 5 per cent". Its truth lives in a document that persists, and
anyone can check it on any day. The office's verdicts were built for this kind, and it is good at them.

An **observation** says what the page's author did and saw: a candidate *tested in code and found
false by nine letters*; a census that *found 334 pages printing a command, none broken, as this
ships*; a provider that *answered with this message on 26 September*. Its truth lives in an event that
is over. The evidence is
whatever record the author kept, and the possibility of making the observation again from the
foundation: the same code at the same commit, the committed data, git history, or (for a model's
behaviour) the same kind of model.

So an observation is checked in three steps, in order:

1. **The record.** A log line, a committed output, a commit message, a dated capture. If the record
   describes the observation, the claim is supported, even when the raw output is not kept.
2. **The foundation.** If the observation can be made again from the repository, and it is cheap,
   make it. If the result disagrees, find out whether the conditions were the same before concluding
   the page was wrong.
3. **Testimony.** If there is no record and it cannot be made again, it is an observation, not kept.
   That is a fact about the evidence, not a finding against the page.

And three things are never evidence against an observation:

- **Later data about an earlier state.** A census in August is not contradicted by the corpus in
  September; outputs before a repair are not contradicted by outputs after it.
- **A different object.** If the author tested their recollection of a published sentence, testing the
  published sentence does not test the claim.
- **The absence of something the record was never meant to hold.**

Observations can be wrong, and the office found real ones: `nobody-answers-at-the-old-number` told
readers four models were "interviewed again that night" when no transcript from that night exists
and its own knock file records three provider errors and a single answer; `the-lineage-measured` counted 34 commits because `wc -l` drops a final unterminated
line (git gives 35); `what-the-lens-caught` printed a run of an earlier engine as the current
verifier's output. Those are errors, found in the right way: against the author's own record, or by
re-making the observation.

When a true observation has been overtaken by the world, it is **stale**, not wrong. If the page dates
it, nothing needs doing. If the page presents it as current, date it, or put today's figure beside it
with its own date. A reader takes "Corrected" to mean the page said something false; the honest label
for the second is "Updated".

## The cases

Every row was examined in git history by a forensic agent on a written brief, and the claim that
decides it was re-checked by the investigating session (marked ✓, and most are also in
`evidence.mjs`). Condensed reports: `reports/`.

| layer | what the page reported | how the office filed it | what it was | what the fix did |
|---|---|---|---|---|
| the-fixed-point | its first candidate, "a well-circulated 'This pangram contains…' sentence", tested in code and found false by nine letters | UNVERIFIABLE, then withdrawn: "no record names that sentence, and Sallows's published pangram counts true" | ✓ **a recorded observation, mis-scoped.** `memory/log.md:1762` records the test; the candidate was the author's own from-memory version of the sentence, which the page should have said. Recreated in kind: 4 of 7 from-memory versions (this session's and six fresh contexts) fail the page's own counter, by 3 to 19 letters; one has Sallows's counts exactly and writes "and" for "&", which is enough to make it false by three | withdrew the observation, and the live Corrected line calls it one "which no record supports", which is **false**. Same day, `the-lineage-measured` was corrected to state the same observation as fact |
| the-cold-hand | "cross-checked against brute-force enumeration of all 2ⁿ sequences for n ≤ 24"; two-sided statistic −7.93 pp; "every figure computed client-side" | WRONG | ✓ **mixed.** "Client-side" was a genuine minor error (the two-sided figure and the check ran offline). The enumeration is recorded (`memory/log.md:1509`, "enumeration + DP + MC, all agreeing") and re-runs from the page's own code: 33,554,428 sequences, agreement 1.9e-11. −7.93 is within the spread of the author's Monte Carlo (four runs of 3 million: −7.925 to −7.940); the exact value is −7.9386 | public line says the n ≤ 24 run "is not on record" (**false**) and "(it is −7.94 pp)", which presents a method-dependent last digit as the author's mistake |
| invisible-exactly-in-focus | "mutate.mjs breaks the engine in ten plausible ways", with the run printed | WRONG, its only source the /checks/ URL that returns 404 | ✓ **true and preserved.** `mutate.mjs` run in a scratch copy reproduces the printed output byte for byte ("10 of 10 caught"). The defect is that the file is not published | open note; `/assay/` shows the true sentence in bold as a false claim |
| every-edge-but-one | two new sequence terms, A059103(10) = 91,658 and A350507(10) = 105,680 | UNVERIFIABLE: "no outside source has a 10th term yet" | **a first-hand computation, committed and reproducible.** Code and output are in `research/gp-10-crosscheck/` (✓ the committed count, 92,167, from which the page subtracts 509); a forensic agent rebuilt the count from scratch with nauty in about a minute, and both terms reproduce exactly | none; the record understates it |
| what-two-minds-disagree-about | the 84 differing values "resolve into 14 distinct disagreements", four nobody's | WRONG; fault attributions UNVERIFIABLE because "the pre-repair outputs … are not kept" | the headline was a **genuine error** at birth (its own capstone, committed with it, says so). But the pre-repair outputs **are** kept: git has them (1f2f8e5d5c, a9e9b8fdb2), and the old engines re-run byte-identically. ✓ Rebuilding them also shows a divergence the capstone's list of "every divergence" omits: before its repair, engine B counted no Costas arrays from order 5 up | the headline fix is sound; the UNVERIFIABLE line is wrong, and ✓ the Corrected line cites `research/claims-pass/…`, a path that no longer exists |
| nobody-answers-at-the-old-number | "Two of the four silences say the model is available for money" | WRONG; the founding audit counted it REAL-FACT | ✓ **true when written, left stale by regeneration.** The 26 July probe had 4 silences, 2 priced; the literal survived rebuilds on 08-02 (5 and 2) and 09-23 (10 and 4) | computing the count was the right fix; the public line names only the 09-23 probe, so a reader concludes the page miscounted. Only a code comment says "true of an early probe and of none since" |
| the-check-that-cannot-fail | a pinned "FAIL 110/111"; 334 pages, 426 commands, zero broken "as this ships"; "the negative control held" | five findings "[declined] NOT ADDRESSED … Needs a census re-run, re-pin" | **dated observations, exactly reproducible at their commits** (rebuilt trees give PASS 96/96 at ship and FAIL 110/111 at 08-10, failing exactly the two checks the page names). ✓ The evidence file behind the last two was overwritten on 2026-08-25 by an unreviewed heartbeat of another wave (69 layers became 24). Two genuine slips in the dek | none yet. Under today's gate a WRONG verdict forces a fix, and the mechanical one would put another experiment's numbers under the old prose |
| entity-at-the-terminal | "answers only in ASCII art, never a word of prose" | WRONG, from live replies captured 2026-09-28 | **drift in a live system**: the models behind the page changed after it was written (the first model in its June chain left the catalogue in July), exposing an absolute "never" about unguarded model output | the guard was right; the Corrected line says the promise "was false in practice" without saying since when |
| the-number-has-no-date | $1.25 billion and $572.1 million "for the December 17, 2025 drawing" | WRONG | **a real advertised estimate** (✓ Wayback, the Powerball homepage at 20:01 UTC on 17 December 2025: "Estimated Jackpot $1.25 Billion", "Cash Value $572.1 Million") attached to the draw rather than to the advertisement | a model fix: "as advertised before the drawing", with the final figures beside it, and no Corrected line |
| how-long-do-ai-models-last | a provider's message, "verbatim", on 26 September | fixed: "filled out from another day's record" | **an observation whose logger cut it at 90 characters** (documented at birth); the relay returns the full text, and the same text is recorded on four other dates. "Filled out from another day's record" is an inference, not established | the public line is accurate; the record says more than the evidence |
| after-hours, held-in-common, how-you-know, the-machine | "the page makes no third-party requests" | WRONG | ✓ **true when written**: all four sentences are in the 2026-06-05 root; the host began injecting its analytics beacon on 2026-06-09 (7d295c248a). ✓ The human ruled on 2026-09-28 that such sentences stand, read through `/privacy/` | the four verdicts are still WRONG on `/assay/`; two further pages still carry the disclaimer sweep the ruling said not to do |

### Handled right, and worth copying

- **the-order-that-was-never-a-schedule.** A live endpoint now returns 78 records where the page says
  77. The pass declined it: "not an error … the page states its retrieval date." The raw data is
  committed with hashes, so the auditor could diff by id and call it drift. What separates this from
  the cases above is not truth; it is a preserved, dated observation.
- **eight-minutes-of-arc.** The author observed that Horizons refused Mars before 1600 (true of body
  499) and wrote "it has no Mars before 1600" (false of the barycentre, 4). The fix kept the
  observation and re-scoped it: "Its Mars body, 499, also stops at 1600, though the Mars barycentre,
  4, reaches further back."
- **a-sextillion-ways-home.** 85 million annealing steps never beat width 23; the page had called 23
  "intrinsic". The fix kept the run and re-scoped the inference to an upper bound.
- **phossy-jaw.** "302 of 302 quotes" cannot be re-run by a reader; declined, because the page does not
  claim a reader can.

## The birth-error claim

`research/claims-audit/RESULTS.md` says of the founding audit: "For 15 of the 18 control layers with a
REAL-FACT, a distinctive string of the wrong text was located in git (`git log -S`). **All 15 were
introduced in the layer's first commit.**" The list of 15 was not kept. Reconstructed: 17 of the 18
were false when written (16 in the first commit, one in a review commit 45 minutes later). The
eighteenth, nobody-answers, was true when written. `git log -S` finds when a string was written, not
when it became false, so the method could not have told them apart. The main point survives, at 17 of
18; the sentence does not, as written. (Also: HEAD has a single squashed root, 0982757349 of
2026-06-05, so for anything older `-S` on HEAD can only answer "first commit"; the true births are on
old `origin/claude/*` branches.)

## Why it happens

- **The verdicts presuppose a source outside the claim.** An observation's source is its record or its
  re-making, and neither is named in the brief. So "no outside source" becomes UNVERIFIABLE, and "no
  record of the exact bytes" becomes "unsupported".
- **"Stale" has no verdict and no disposition.** A figure true when measured and overtaken since can
  only be WRONG (which since 2026-09-28 must be fixed), MINOR, or declined against the definition of
  declined.
- **Every change is labelled "Corrected".** The label carries a charge the author does not always owe.
- **Evidence is overwritten.** Two of the observations above were preserved and then clobbered by later
  commits (an unreviewed heartbeat checkpoint of another wave; repairs that rewrote their outputs in
  place). Git kept both, which is why they could be rebuilt, but nothing told the auditor to look there.
  (The shared mutation record was fixed on 2026-09-06 to merge rather than replace, which stopped
  further clobbering but did not bring the 66-layer record back: the file held 26 layers on
  2026-09-28.)
- **Rulings do not reach records.** The beacon verdicts were not revised after the human's ruling.

## What follows (proposals; none adopted here)

1. **In the brief** (`oversight/claims-pass.md`): the three-step check for observations and the three
   things that are never evidence against one, as above.
2. **In the vocabulary and the gate** (`scripts/check-claims-pass.mjs`): two verdicts. `OBSERVED` (the
   record, the re-made observation, or "not kept"), and `STALE` (true when observed, overtaken since),
   answered by `[dated]`, never counted in `wrong`, never written up as a correction. The second-pass
   classes (R, I, D, N) gain one for staleness, so it is not charged to either pass.
3. **On the pages:** "Updated" for a dated figure brought up to date, "Corrected" only for what was
   false when written.
4. **Repairs:** the-fixed-point (restore the observation, re-scoped; remove the false sentence),
   the-cold-hand (the run is on record; the last digit is a method difference), invisible-exactly-in-focus
   (publish `mutate.mjs`; the verdict), every-edge-but-one and what-two-minds (the UNVERIFIABLE lines),
   nobody-answers (say the sentence was true of the July probe), the-check-that-cannot-fail (restore the
   July evidence under a dated name, date the pin, fix the dek), the four beacon records, the two pages
   still carrying the beacon sweep, and the birth-error sentence in the claims audit.
5. **For evidence:** a page's observations should live in files that are never overwritten, named by
   date, the way `research/second-space/census/probe-<date>.json` already are.

## Limits

About twenty cases were examined forensically, chosen because their record text suggested an
observation; the other ~70 records were read, not re-derived. The forensic agents and the blind
classifier are Claude, as are the auditors they check, so their blind spots may be shared. The
investigating session adjudicated knowing the hypothesis; every adjudication names its evidence so it
can be checked without trusting it, and `evidence.mjs` re-derives the load-bearing ones. The pangram
recreation shows the phenomenon recurs, not the June instance's exact candidate, which is gone.

## Files

- `evidence.mjs`: the checks.
- `lines.jsonl`: every claim, finding and against-the-first-pass line from `assay/records/` and
  `assay/second/` as of 2026-09-28.
- `preclassification.md`: the investigating session's own flags, written before the forensic reports
  and the blind classification arrived.
- `classified.csv`: the blind classification of every non-confirmed verdict and declined finding.
- `reports/`: the forensic reports, condensed, with what the investigating session re-verified.

## What was done (2026-09-28, after the human's decision)

**Rules** (landed at 74c2d55e80): the brief's step 7 (the record, then the foundation, then
testimony; never later data about an earlier state, a substituted object, or the absence of what a
record was not meant to hold); OBSERVED and STALE verdicts in `scripts/check-claims-pass.mjs`, with a
`[dated]` disposition, never counted in `wrong`; a second-pass class, [O] overtaken; "Updated", not
"Corrected", for a figure true when written; the office's reader and `/assay/` pages show the new
counts and keep a page's Updated dates apart from its corrections; the "every error was there at birth"
sentence corrected in `assay/README.md` and `research/claims-audit/RESULTS.md`, 17 of 18; one sentence
in WAKE.md. Every existing record still passes; selftest 28/28.

**Repairs**, each re-checked by its page's own verifier or the gate:

- the-fixed-point: the observation restored, scoped to the author's recollection; the false "which no
  record supports" replaced, with a dated note; the record's line is OBSERVED.
- the-cold-hand: the correction no longer says the n ≤ 24 run is not on record, or sets a last-digit
  method difference against the author as an error; the check paragraph reports the n ≤ 24 re-run.
- nobody-answers-at-the-old-number: its silences line reads "Updated", saying the sentence was true of
  the probe of 2026-07-26 (generator and page); the record's line is STALE; the second pass's catalogue
  line is OBSERVED against its dated roster. Verifier 74/74.
- invisible-exactly-in-focus: the page names `mutate.mjs` by its path, so it is published on /checks/
  (byte-identical); the record's line is OBSERVED; note 34dc8d closed. Verifier 44/0.
- every-edge-but-one, how-long-do-ai-models-last: OBSERVED, with their records.
- what-two-minds-disagree-about: OBSERVED against the outputs git keeps; the correction's dead link
  fixed. Board note 243b68 holds the Costas divergence the capstone omits, for a pass on that layer.
- entity-at-the-terminal: the correction says which model led the chain when the promise was written.
- the beacon: the four verdicts are CONFIRMED under the human's ruling (and true when written); the two
  pages that still carried the disclaimer now read as the other pages do, with Updated lines.
- the-check-that-cannot-fail: the mutation sweep its figures came from is restored as
  `research/verifier-independence/dated/mutation-results-2026-08-10.json` and read by its verifier and filler,
  so the negative-control check passes again; an Updated line says the page's figures are the census of
  2026-08-10, with the recount of 2026-09-28 beside them; the dek's "Four bugs" corrected to five and its
  595 dated. The re-measure of this living census is board note 84ccb1. Its record now carries three
  STALE lines where it had five "declined, not addressed".

**Sweeps**, pre-registered before any agent ran (`SWEEP-PREREG.md`), results in `SWEEP-RESULTS.md`.
Six agents read the 37 layers with a public correction line not examined above (211 items) and the 13
flagged lines left over (sweep 2). Most corrections are right: 195 of the 211 items were false when
written. After my re-check, 6 of the 37 layers needed a repair, and none of them was staleness: two
withdrew an observation their record supports (coloring-the-plane's "three ways"; no-field-for-the-repair's
13,493 bridges, where the fix put a subset count in the true one's place and cited "the data" for it),
and four were correction lines that misstate the evidence. No step checks a correction line against
the record before it goes up (a later reading caught one of the four); that is the sweeps' addition. Of the 13 flagged
lines, 9 were true when written, recorded or reproducible, and for five of them only the record needed
to change. Nine proposed repairs I declined, with reasons. Every repaired page that has a verifier
passes it.