# Pre-registration: can a published machine measurement be re-run? Written and committed **before any probe or re-run data was collected**, so the classification rules and the comparison cannot have been chosen after seeing the answer. Git holds the timestamp; `git log --diff-filter=A -- research/model-arm-reproduction/PREREGISTRATION.md` shows when this file entered the repository relative to everything in `data/`. ## What already exists, and is not in question On 2026-08-20 and 2026-08-21 the Artificial Wasteland's living-experiments wave sampled a panel of language models for six strata. Those runs are published: | stratum | generated | recorded draws | panel | |---|---|---:|---:| | anchoring-rct | 2026-08-20 | 600 | 6 | | calibration-intervals | 2026-08-20 | 450 | 6 | | guess-two-thirds | 2026-08-20 | 300 | 6 | | serial-position | 2026-08-20 | 360 | 6 | | the-coin-you-cant-fake | 2026-08-20 | 300 | 6 | | wason-246 | 2026-08-21 | 352 | 6 | Total 2,362 draws. The panel was not chosen by hand: `run-model-arm.mjs` holds a fixed ten-id candidate list and admits the first six that answer a one-line probe. The protocols are frozen in `research/living-experiments/model-arm/requests/.json`, the results in `.../results/.json`, and each page's own sealed `analysis.mjs` turns draws into the numbers it prints. ## The question Seventeen days later, on 2026-09-07: **can that measurement be re-run?** Split into three, because they have different answers and the difference is the point: 1. **Does the panel still exist?** Of the ten candidates the published method draws from, how many can be reached today, and by what evidence? 2. **Where a model still answers, does it still answer the same?** Same id, same frozen protocol, same temperature, seventeen days apart. 3. **Could a reader assemble a panel at all today?** The method needs six of ten. ## Instruments, and why there are two - **The public catalogue.** `https://openrouter.ai/api/v1/models` needs no key and no account. For each id it says whether the id is listed and what its prompt price is. This is the check a reader can perform. A snapshot is committed at `data/openrouter-catalogue.json`. - **The rail.** The site's own key-gated `/api/collective/model-run`, using the original harness's probe verbatim: user `Reply with exactly the word: ready`, temperature 0, `max_tokens` 300. This says whether the model actually replies. The two can disagree, and the disagreement is the finding worth having: an id listed on the free tier that will not answer is invisible to any citation, any catalogue check, and any reader who does not actually call it. ## Classification, fixed here in advance Per candidate, over K ≥ 6 attempts spread across rounds separated in time (so a transient outage cannot masquerade as a withdrawal), with every attempt kept verbatim in `data/probe-attempts.jsonl`: | verdict | rule | |---|---| | `answered` | at least one attempt returned `ok:true` with a non-empty answer | | `listed_but_silent` | no attempt answered, **and** the id is in the catalogue with prompt price `0` | | `listed_paid_only` | no attempt answered, the `:free` id is absent, **and** the bare base id is listed at a non-zero price | | `delisted` | no attempt answered, and neither the `:free` id nor a base id is listed | One failed attempt is never a verdict. A model that answers once is `answered`, however badly it answers: this measures reachability, not quality. ## The re-run, and what counts as a comparison For every candidate classed `answered`, the frozen protocols of the six slugs are re-sent through the same harness, unmodified, on 2026-09-07. Then, per slug, the comparison is made **by the page's own sealed `analysis.mjs`**, the same function that produced the published number, over the 2026-08 draws and the 2026-09 draws separately. We do not write a second analysis: a re-run scored by a new parser would compare two things at once. Declared before the data: a difference is **reported as measured, not as significant.** Each model's per-protocol draw count is small (25 for most protocols, 1 for the 60 serial-position lists), the models are sampled at temperature 1, and two draws of the same prompt from the same model differ by construction. So the pre-committed statement is a paired description (the published value, the re-run value, the difference, and the count each rests on) and **no significance claim is made about any individual model's shift** unless it is accompanied by an interval computed here and shown. ## What would make this study wrong, and how a reader checks - **If the "silent" models are merely having a bad hour.** Mitigated by rounds spread in time and by keeping every attempt; a reader can re-run `--classify` over `data/probe-attempts.jsonl` and see the timestamps. - **If the rail, not the provider, is the thing that broke.** Falsified by the same rounds containing successful calls to other models through the same rail at the same minute. - **If the catalogue snapshot were cherry-picked.** The whole 430-row snapshot is committed, not just the rows used, and the live URL is public: anyone can fetch it today and compare. - **The one thing a reader cannot repeat:** the rail is key-gated, so the reachability probe itself is not reproducible from outside. The catalogue half is, and it is the half that carries the delisting claim. The silence claim rests on our committed attempt log, and is stated as such. ## Standing hazards - Free-tier availability is a commercial arrangement, not a property of a model. Nothing here says a model was deleted, retrained, or changed. It says it could not be reached, on this rail, on this date, and what the public catalogue said at the same moment. - OpenRouter routes an id to providers that can change underneath it. A same-id difference is therefore not necessarily a changed model; it may be a changed route. This is a limit on the *interpretation* of question 2, and it is exactly the limit a citation to a hosted model id also has. --- ## Amendment 1 (2026-09-07, during collection, before any reordered slug had a re-run figure) **Sampling order corrected, on a property of the August data.** The plan sampled slugs in descending order of "how cleanly a slug could be compared", and named `the-coin-you-cant-fake` first because it is the only slug carrying a temperature-0 protocol. That guess was wrong, and counting the published arms rather than reasoning about them showed it: the coin's parser accepts only an exact 100-character string of `H` and `T`, and across its 300 published draws it classified **seven** as usable, six of them from a single model. `anchoring-rct`, which the old order sampled last, classified **453 of 600**. The order is now set by usable classified draws in the **published** arm, which is a fact about August and cannot be influenced by any re-run outcome: | slug | usable / draws, published arm | |---|---:| | `anchoring-rct` | 453 / 600 | | `wason-246` | 249 / 352 | | `serial-position` | 203 / 360 | | `calibration-intervals` | 193 / 450 | | `guess-two-thirds` | 69 / 300 | | `the-coin-you-cant-fake` | 7 / 300 | Draws already collected under the old order are kept, not discarded: they were sampled under the same frozen protocols and the harness resumes rather than re-asks. Nothing else changes. **A pooled row is added, and it is restricted.** Per-model, per-protocol cells in this wave are small enough that a per-cell difference says almost nothing. A pooled figure has enough n to look at, but pooling across the whole roster would confound the two things this study exists to separate, since the published run pooled six models and the re-run can reach four. So the pooled row is computed on **both** sides over the intersection of the two rosters, using the same page function. It answers the narrower and answerable question: holding the panel to the models that still exist, did the number move?