Loading the measured data.
Four of ten, where six are needed
The panel was never hand-picked. run-model-arm.mjs holds a fixed list of ten free-tier model ids and admits the first six that answer a one-line probe. That rule is why this question has an answer rather than an opinion: the method is a program, so it can be run again and watched failing.
·
Reachability measured · over eight attempts per model, spread across four rounds separated in time, every attempt kept. Catalogue snapshot taken ·.
The ten, in the order the method reads them
Two instruments, asked the same question. The catalogue is OpenRouter's public model list, which needs no key and no account: it says whether an id exists and what it costs. The rail is an actual call, which says whether the model replies. They disagree, and the disagreement is the part worth having.
| # | model id | today | answered | the public catalogue says | most common failure, verbatim |
|---|
Listed, free, and silent
Both Google ids sit in the public catalogue at a prompt price of exactly 0, and returned Provider returned error on all eight attempts. This is the case that no citation format records and no availability check catches. A reader who does the sensible thing, looks the model up, and finds it listed and free, concludes the study reproduces. It does not. The listing is a commercial statement about an arrangement; it is not a promise that anything is running.
The three ids marked free access withdrawn fail more honestly. The rail returns, word for word, This model is unavailable for free. The paid version is available now, and names the paid id to use instead. The model still exists. What ended was the arrangement under which we measured it, and a measurement made under a free tier is not repeatable by paying, because the thing behind the paid id need not be the thing that answered.
It had already started, and the published files say so
The decay is not something that happened to the study afterwards. It was happening while the study was being made, and the August record contains the evidence without anyone at the time reading it that way.
google/gemma-4-31b-it:free was rostered for the first tranche on 20 August and answered 2 of its 100 anchoring draws and 1 of its 50 guess-two-thirds draws. By the very next tranche, hours later, it failed the roster probe outright and was dropped with the reason Provider returned error, alongside its Gemma sibling, and a different model took the sixth slot. That is why the wave's six layers do not all share a panel: the roster changed inside the run. Seventeen days later the same two ids give the same error, still at price zero, still listed.
So the honest description of what a citation to a hosted model buys you is not that it will expire. It is that it may already have been expiring while you wrote it down.
Check it yourself, from where you are
The half of this that rests on our own key is only half. The delisting and pricing claims rest on a public endpoint that anyone can read, and it sends access-control-allow-origin: *, so your own browser can ask it the same question right now and compare the answer with ours.
One button, one third-party request
Pressing this fetches https://openrouter.ai/api/v1/models from your browser. It is the only request this page makes to anyone but this site, it happens only if you press, and nothing about you is sent beyond what any browser sends when it asks for a public file.
Or from a terminal, with nothing installed but curl and Python:
curl -s https://openrouter.ai/api/v1/models | python3 -c "
import json,sys
ids = {m['id'] for m in json.load(sys.stdin)['data']}
for c in ['google/gemma-4-31b-it:free','openai/gpt-oss-20b:free',
'nvidia/nemotron-3-ultra-550b-a55b:free','nvidia/nemotron-3-super-120b-a12b:free',
'poolside/laguna-s-2.1:free','nvidia/nemotron-nano-9b-v2:free',
'google/gemma-4-26b-a4b-it:free','poolside/laguna-xs-2.1:free',
'meta-llama/llama-3.3-70b-instruct:free','inclusionai/ling-3.0-flash:free']:
print(('listed ' if c in ids else 'ABSENT ') + c)"
If you run this long enough after 7 September 2026, you will get a different answer than we did. That is the finding, not a defect in it.
What came back from the four that answered
For every model still reachable, the frozen protocols were re-sent unchanged: same prompts, same temperature, same token budget the published run used, read from the same request files, whose hash is recorded beside the new draws. Then both runs, the old and the new, were scored by the published page's own analysis module rather than by anything written for this study. A re-run scored by a second parser would be comparing two things at once, and no difference could be attributed to either.
The result that was not expected
A hosted measurement can fail to reproduce in two ways. The model can answer differently, or the model can stop being askable. Seventeen days of this wave produced a great deal of the second and, on the arm with the most usable data behind it, almost none of the first.
Look at anchoring-rct below, which is the comparison to look at first because the published arm classified 453 of its 600 draws as usable, against seven of three hundred for the coin. Held to the models that still exist, its median estimates come back exactly equal on both items, to the digit, seventeen days apart. Not close. Equal.
serial-position says the same thing from a different direction and with a great deal of n behind it: pooled over the shared panel, the exact-match reporting rate is 0.9914 across 2,085 word positions in August and 0.9877 across 2,025 today, and two of the four models sit at exactly 1.0000 on both dates. On calibration-intervals the two models with real counts move by 0.015 and 0.006.
Read all of that with its caveat attached, because it is doing less work than it looks. The Houston item's median is 2,099,451, which is the true 2010 census figure: the models are reciting a fact rather than producing an estimate, and a recited fact is the easiest thing in the world to reproduce. The noisier arms move more, and wason-246's categorical shares move by tenths on cells of a dozen draws, which at those counts is what noise looks like. Still, the direction of the evidence here is one worth naming: what decayed was access, not behaviour.
· paired figures below. The counts matter more than the differences: most protocols draw 25 samples at temperature 1, so two runs of the same prompt on the same model differ by construction. Nothing here is offered as a significance test, and where a cell reads zero the model answered nothing that its page's parser could classify, which is itself the comparison. A model that appears under one slug and not another was never rostered for that one: the wave's own panel already differed between its two tranches, because two models started failing between them.
Where the re-run stopped, and why. It did not finish because it ran out of models to ask; it finished because the account ran out of free calls for the day. Both remaining samplers stopped cleanly on OpenRouter's own Rate limit exceeded: free-models-per-day-high-balance, which is the same tier whose availability this page is about. Slugs were sampled in descending order of how much usable data the August arm actually held, so what the cap truncated is the thinnest end, and every count below is the count that was collected rather than the count that was wanted.
One trap in the pooled row, named because it caught us. Pooling is restricted to the models present in both runs, which removes the obvious confound. It does not remove a subtler one: a model's yield can change even when its answers do not, and a pooled figure then moves because the mix of who contributed usable draws moved. calibration-intervals is the live case below. Its pooled coverage rises sharply, and the per-model rows show why: the two models with poor coverage contributed far fewer classifiable draws this time. So read the per-model rows first and the pooled row second, and never the pooled row alone.
The column that did not decay, because nothing was in it
Each of these layers was built with three arms: a published human study, a machine panel, and a live one that readers fill by taking part. On 7 September 2026, seventeen days after the wave opened, all eleven of those living arms held exactly zero rows.
An empty arm and a broken one look identical from outside, so the second was ruled out before the first was claimed. research/model-arm-reproduction/prove-intake-works.mjs drives one of those pages in a real browser with the store stubbed and asserts eight things, of which the one that matters is that the body posted is byte-for-byte the payload the reader was shown. It passes 8 of 8. Separately, a row was added to the wave's own wiring-check arm through the live rail, whose registry entry says in writing that its value carries no meaning, and the count moved. The path works.
So the two halves of the same wave failed in opposite directions in the same fortnight. The machine arm was easy to collect and is already partly unrepeatable. The human arm is repeatable forever and has never been collected once. It is worth saying plainly rather than quietly: the arms are open, the submission path works, and nobody has come.
If you would like to be the first row in one of them, they are here: optional stopping, false consensus, wisdom of crowds, change blindness, serial position, the focal point, and the six named in the sources below. Each takes under a minute and stores a fixed tuple of bounded integers: no text, no identity, no time finer than the day.
What this is, and what it is not
Not a new claim
That hosted and proprietary models break reproducibility is argued in the literature and this study does not pretend to discover it. What it adds is narrow: a fully open worked instance, where the published measurement, its raw completions, its frozen prompts, its selection rule and its scoring code were all in hand before the reproduction was attempted, so the failure can be watched rather than argued about.
Not a claim about the models
Nothing here says a model was deleted, retrained or changed. It says an id could not be reached, on one rail, on one date, and reports what the public catalogue said at the same moment. A router also sends one id to providers that change underneath it, so a same-id difference may be a changed route rather than a changed mind. That limit is exactly the limit any citation to a hosted model id already has.
The half you cannot repeat
Whether a listed model actually answers was measured through this site's own key-gated rail, so a stranger cannot re-collect those attempts. Every one of them, successes included, is published verbatim beside this page, and the classification is re-derivable from those bytes with no network at all. But the honest statement is that this half rests on our record, and no wording changes that.
Written before the data
The rules were fixed and committed before any datum was collected: what counts as unreachable, how many attempts before a verdict, which quantity would be compared, and a standing refusal to make a significance claim about any individual model's shift. The commit carrying them is older than every file they govern. It is published here as preregistration.md, amendment included, and the amendment is the part to read: the sampling order was corrected mid-run, on a property of the August data that no re-run outcome could influence, and it says so with its reason.
The apparatus
Everything this page displays comes from files sitting beside it, copied verbatim from the study directory. The page computes no statistic of its own; a page about reproducibility that recomputed its own headline in the browser would be a worse offender than the thing it describes.
- roster-status.json the classification, derived from the two files below
- comparison.json the paired figures, then against now
- probe-attempts.jsonl every reachability attempt, verbatim, successes included. This is the whole evidence for the silent rows, and it is here rather than summarised because a conclusion without its working is what this page is about.
- openrouter-catalogue.json the full catalogue snapshot, all 430 rows, not only the ten this page uses
- preregistration.md the rules, fixed before any of the above existed, with the one amendment and its reason
The programs are published verbatim to read: research/model-arm-reproduction/probe-roster.mjs measures reachability and re-derives the classification from the attempt log with no network at all, research/model-arm-reproduction/rerun.mjs re-sends the frozen protocols, research/model-arm-reproduction/compare.mjs scores both runs with the published pages' own modules and carries a positive control that requires every adapter to be deterministic and non-empty, and research/model-arm-reproduction/merge.mjs refuses to join two shards that disagree about which frozen request they sampled.
The classification, re-derived in an empty directory
One of them you can actually run, and it is the one carrying the claim that rests on our own record. Four files, no key, no network once they are down, and it prints the same table this page renders:
mkdir -p aw/research/model-arm-reproduction/data && cd aw curl -sL -o research/model-arm-reproduction/probe-roster.mjs \ https://artwaste.land/checks/research/model-arm-reproduction/probe-roster.mjs curl -sL -o research/model-arm-reproduction/data/probe-attempts.jsonl \ https://artwaste.land/strata/the-panel-that-cannot-be-assembled/probe-attempts.jsonl curl -sL -o research/model-arm-reproduction/data/openrouter-catalogue.json \ https://artwaste.land/strata/the-panel-that-cannot-be-assembled/openrouter-catalogue.json curl -sL -o research/model-arm-reproduction/data/allowlist.json \ https://artwaste.land/strata/the-panel-that-cannot-be-assembled/allowlist.json node research/model-arm-reproduction/probe-roster.mjs --classify
The fourth file is the wider candidate list the probe reads, extracted from this site's worker into plain JSON so that no part of the worker beyond that one array travels with it. Everything the classification asserts comes out of the attempt log and the catalogue snapshot; once the four files are down, the program contacts nothing.