# The error they share: transcript, item bank and results

The complete record of the forced-choice experiment on
[/strata/the-error-they-share/](https://artwaste.land/strata/the-error-they-share/): **595 rows**
in `asks.jsonl`, one per attempt to put one of 21 checked misconceptions to one of 15
free-tier models, with every reply kept verbatim; the 21 questions in `items.json`, each
with its options and the quotes that ground them; and every number the page states, in
`results.json`.

The run took place on 2026-08-14, between 14:38 and 15:48 UTC, through this project's
`/api/probe` route (in `worker.js`), which calls OpenRouter. The findings, and what they do
and do not show, are on the page. This README describes the files.

## The files

| file | what it is | shape |
|---|---|---|
| `asks.jsonl` | the transcript: one JSON object per line | `schema.json`, every row validated |
| `items.json` | the item bank | one JSON array of 21 objects |
| `results.json` | every computed number | one JSON object, generated by `score.mjs` |
| `schema.json` | JSON Schema (draft 2020-12) for a row of `asks.jsonl` | |
| `sources.json` | where each field came from, and under what licence | |
| `manifest.json` | byte length, sha256 and row count of every file | |
| `validate.mjs` | dependency-free validator: `node validate.mjs` | |

## asks.jsonl

595 rows, of four kinds. `schema.json` requires each row to be exactly one of them.

| kind | rows | marked by |
|---|---|---|
| answered | 525 | a `raw` field holding the reply, verbatim |
| upstream fault | 60 | `upstream: true`, with the error text in `detail` |
| silent | 9 | `silent: true`, with the reason in `detail` |
| model not served | 1 | `unserved: true`; a roster note, not an ask |

The 594 ask rows split 299 in arm A (the myth among the options) and 295 in arm B (the
myth replaced by a third distractor). Every ask row records the question, the four
options in the order shown (`shown`), the role of each (`shownRoles`), the position of the
checked answer (`checkedSlot`), and the `seed` from which that order is recomputable. An
upstream fault or a silence is kept, never filled in, and never scored as the model's
answer; where the first attempt at a question failed and a later one succeeded, both rows
are here.

No row records which option a model chose. That is decided afterwards, from `raw`, by the
parse rule written at the top of `research/myth-consensus/score.mjs`, so anyone can apply
the rule themselves or a different one.

## items.json

21 objects. Each has `id` (the value of `item` in the transcript), `stratum` (the page on
this site that checked the claim), `question`, `options` (the texts for `checked`, `myth`,
`d1`, `d2` and `d3`), `evidence` (for `checked` and `myth`, a verbatim `quote` and the
`file` in this repository it comes from) and `mythEvidence`. That last field is `held` for
18 items, where the cited page documents the myth as a belief people hold, and `corrected`
for 3, where the page only corrects it.

The questions, options and distractors were written by this project for this experiment.
The evidence quotes are from this project's own pages.

## results.json

One object written by `score.mjs` (its `generated` field is 2026-08-14T15:50:46Z). Its keys
are `generated`, `parseMode`, `items`, `models`, `tally`, `accuracy`, `errorCompositionA`,
`perItem`, `signTest`, `bootstrap`, `neutralCompositionB`, `pull`, `ensemble`,
`perModelRate`, `makers`, `position`, `chosenSlots`, `checkedSlotRealised`, `votes`,
`flagged` and `sensitivityR1Only`. The page's numbers are read from this file, and the
layer's verifier re-derives them from `asks.jsonl` independently.

## Licences, and one that is not established

The questions, options, protocol fields and results are this project's work, under the
site's CC BY 4.0 licence (see `sources.json`).

The replies in `raw`, and the error text in `detail`, came from the models' makers through
OpenRouter. OpenRouter's terms leave the rights in a model's output to each model's own
terms, and those were not read for this record, so `sources.json` says **not established**
for them rather than guess. The data room's ORIGIN condition fails this dataset on those two
fields as a result, and that failure is accurate.

## How it is built

`research/myth-consensus/ask.mjs` wrote `asks.jsonl`; `score.mjs` wrote `results.json`;
`items.json` is the committed item bank, whose quotes `check-items.mjs` finds in the files
they cite. These files are copied
here byte for byte by `research/data-room/emit/the-error-they-share.mjs`, and
`node research/data-room/publish.mjs --check` proves the published bytes are the committed
ones.