# Hansard Reaction Notation Concordance

This dataset is a concordance of reaction-like notation extracted from the UK parliamentary record. Its rows contain such things as reporter-written laughter, cheers, interruptions, cries, and generic-member interjections, plus a short preceding context and source provenance. It is designed to be used independently of the Artificial Wasteland page.

The unit is a notation cue in the published record. It is not an acoustic event, a joke, a verified quotation, or evidence that adjacent words caused a reaction.

## Files

- `cues-YYYY.jsonl`: one JSON object per line, sharded by the year written in the source path or filename.
- `schema.json`: JSON Schema draft 2020-12 for each cue row.
- `sources.json`: acquisition endpoints, URL resolution rules, digest status, retrieval-time status, and licence status.
- `validate.mjs`: a dependency-free validator and per-field null-rate report.

Empty year shards are valid. They mean the frozen predicate returned no rows for the material scanned, not that no reaction occurred.

There are 222 shards for the 224 years from 1803 through 2026. The two missing years, 1816 and 1829, were in the scan, but their sitting-day index listed no days, so there was nothing to sample. The stored sampling plan (`_victorian-plan.json`) lists both years with `sitting_days_in_year: 0`, and the historic harvester opens a year's shard only when it scans at least one sampled day, so neither year got a file. Because the harvester's fetch returns an empty page after three failed attempts, the plan alone cannot tell an empty index from a failed request. A fresh check settles it for the source as it stands now: on 2026-09-26 `https://api.parliament.uk/historic-hansard/sittings/1816` and `.../sittings/1829` resolved, after a redirect, to HTTP 200 pages with no month links, `.../commons/1816`, `.../lords/1816`, `.../commons/1829` and `.../lords/1829` likewise listed no months, and the decade pages `.../sittings/1810s` and `.../sittings/1820s` link every other year of their decade but not these two. This is a gap in the digitised archive's index. It is not evidence that neither House sat in those years, and those years have no defined inclusion rate. (Added 2026-09-26.)

## Frozen predicate

The predicate was fixed before counting. Matching is case-insensitive.

For both regimes, a bracketed span must be 2 through 200 characters before markup flattening and must contain a match for this regular expression:

```text
(laugh|interrupt|cheer|hon\.?\s*mem[bk]|cries?\s+of|hear,?\s*hear|ironical|uproar|disorder|noise|oh[,!.]|shame|withdraw|order[,!.]|applause|groan|jeer)
```

For ParlParse XML only, a second route includes an entire `<speech>` element when its `speakername` starts with an optional `a` or `an`, followed by `hon`, an optional full stop, whitespace, and `mem` followed by `b` or `k`. It also includes an exact case-insensitive speaker label of `Hon. Members` or `An Hon. Member`. These rows have `kind: "attributed_speech"`.

This deliberately loose `mem[bk]` fragment admits OCR forms such as `MEMBEKS`. OCR corruption is evidence about digitisation and is not silently repaired.

## Two acquisition regimes

### 1803 through 1918: seeded sample

This regime samples up to 15 sitting days from every source-labelled year, using seed `20260903 + year`. It scans all section links discovered for the Commons and Lords on each selected day. The sampling frame is the sitting-day index returned by `api.parliament.uk`; the inclusion rate for a year is `min(15, N) / N`, where `N` is the number of indexed sitting days in that year. Years with no indexed days have no defined rate. The stored sampling plan is `_victorian-plan.json` in the research archive, not a census.

The sample has sampling uncertainty, unequal rates between years, and possible coverage error in the archive indexes. Do not treat raw yearly sample counts as directly comparable totals.

There is a known, severe source-date fault in this regime. For example, the source URL labelled `22 August 1804` contains `Navy Estimates, 1894-5`, a Secretary for Scotland, and an 1890 Act. The public row preserves the source path in `date`; it does not certify that date. Chronological analysis of the sampled regime requires an independent date audit first.

The 1909 adoption of the Official Report is another hard seam inside the sampled era. Analyses must mark it as a break, not draw one uninterrupted 1803 to 2026 trend.

### 1919 onward: XML census

This regime scans every file named by the mySociety ParlParse Commons debates XML directory. At the harvest described by the protocol, the directory listed 20,084 files from 4 February 1919 through 2 September 2026. Call a release complete only if its harvest summary reports all listed files scanned and an empty failure list. This is a census of files available through that collection and predicate, not a census of chamber sound or every kind of parliamentary proceeding.

The seam between the HTML sample and XML census is 4 February 1919. The regimes have different source formats, discovery rules, fields, and coverage. Compare them only as separately labelled series.

## Exclusions and transformations

The following are outside the dataset or changed by extraction:

- A bracketed span is excluded if it is shorter than 2 or longer than 200 characters in the matched source markup.
- A bracketed span is excluded unless it matches at least one frozen term above. The broad terms can also create false positives.
- Unbracketed reactions are excluded, except for XML speeches admitted by the generic-member speaker predicate.
- Unbracketed interjections attributed to a named member are excluded by the generic-member route.
- XML outside lowercase `<speech>...</speech>` matches is excluded. Attributes must use the double-quoted form recognized by the harvester.
- XML sources outside the ParlParse `scrapedxml/debates/` listing are excluded. This omits written answers, many committees, and other parliamentary collections.
- Historic HTML is limited to Commons and Lords section links discovered beneath sampled sitting-day pages. Other page types and undiscovered links are excluded.
- A historic day is excluded when its index does not contain the literal marker `Sitting of`.
- A source request is abandoned after three failures. The XML harvester reports failed filenames in its final summary. The HTML harvester does not retain a failure ledger, so its missing pages cannot be distinguished from pages with no matches.
- Markup is removed and whitespace is collapsed. Only a short, fixed list of character references is decoded. Other references can remain visible in text.
- `cue_text` is truncated to 300 code units for bracketed rows and 400 for attributed XML speeches. `preceding_text` is truncated to its final 400 code units.
- Brackets are removed from bracketed `cue_text` values. They are not restored in this release.
- The latest preceding XML `major-heading` is copied into `heading`. Historic HTML headings are not retained.
- Empty source speaker labels remain empty strings. Missing regime-specific fields remain absent and count as null in the validator report.
- Source dates, speaker labels, spellings, entities, and OCR mistakes are preserved rather than silently corrected.
- Re-running either current harvester into a non-empty shard appends duplicates. A release must be made from clean output or deduplicated with that fact recorded.
- The current harvesters do not retain rows they cannot parse. Consequently there is no honest unparsed-row file to ship from this run. Absence from such a file must not be interpreted as zero parse failures.

## Empty strings in XML rows

Some XML fields are present but empty. The counts below were measured over all 224,023 rows on 2026-09-26. The first three were originally forbidden by `schema.json` (`minLength: 1`), and the validator rejected those 14 rows once the shards were published beside it. The schema now allows them and says when each can occur. No value was filled in.

| Field | Empty in | Why |
| --- | ---: | --- |
| `cue_text` | 2 `attributed_speech` rows | The source speech element is `<p></p>` under `speakername="Hon. Members"` (`debates1962-07-10a.xml`, speech `1138.5`; `debates1993-07-07a.xml`, speech `331.2`). The source has no text to copy. |
| `colnum` | 6 `bracketed_cue` rows from 3 speeches | The source `speakername` attribute contains a literal `>` (`"Mr. Bowden>"`, `"Mr. Tony Blair>"`, `"Sir Graham Bright>"` in `debates1964-11-10a.xml`, `debates1984-04-02a.xml`, `debates1996-04-30a.xml`). The harvester reads a speech's attributes only up to the first `>`, so it lost `speakername` and `colnum`. The source does carry columns 965, 692 and 901. `cue_speaker` and `preceding_speaker` are empty in these rows too, and the 1996 row's `preceding_text` begins with the leftover attribute text `" colnum="901" time="">`. This is a harvester defect, not an absence in the source. |
| `preceding_id` | 6 `attributed_speech` rows | The matched speech is the first speech element in its file (`debates2009-02-26a.xml`, `debates2010-01-06a.xml`, `debates2012-07-03b.xml`, `debates2013-01-22a.xml`, `debates2013-12-04b.xml`, `debates2025-06-13b.xml`), and the harvester does not look back into another file. `preceding_speaker` and `preceding_text` are empty in the same rows. |
| `preceding_text` | 6 `attributed_speech` rows, 94 `bracketed_cue` rows | For the attributed rows, as for `preceding_id`. For a bracketed cue, no text precedes the bracket in its speech once markup is removed. |
| `cue_speaker` | 1,098 `bracketed_cue` rows | The containing speech has no usable `speakername` (including the 6 rows above). |
| `preceding_speaker` | 1,098 `bracketed_cue` rows, 8,245 `attributed_speech` rows | For a bracketed cue it copies `cue_speaker`. For an attributed speech the previous non-empty speech had no usable `speakername`, or there was no previous speech in the file. |
| `heading` | 4,224 `bracketed_cue` rows, 59 `attributed_speech` rows | The latest `major-heading` before the speech in its file is missing or has no text. |

All 11 source files named above were refetched on 2026-09-26 and their SHA-256 values begin with the 16 characters stored in the rows' `sha256`, so these are the same bytes the harvester read.

## Provenance

For an XML row, join `file` to the `record_url_template` in `sources.json`. The `speech_id` names its containing `<speech>` element and `colnum` records that element's column. The field called `sha256` is only the first 16 hexadecimal characters of the source-byte SHA-256 because the current harvester discarded the rest. It is useful as a check prefix but is not a full digest.

For a historic HTML row, `url` is the exact section URL and `section` repeats its final path component. The harvester did not record response bytes, a hash, a retrieval timestamp, a column, a speech id, or a bracket offset. Those values are unknown. `sources.json` uses `null` and `UNKNOWN` for them.

These are material provenance limits. A future release should capture full source SHA-256 values, UTC retrieval timestamps, and a within-document locator. It should declare a method break rather than retroactively filling old rows.

## Loading in Python

From this directory, a year shard loads in three lines:

```python
import json
with open("cues-1919.jsonl", encoding="utf-8") as source:
    rows = [json.loads(line) for line in source]
```

For large shards, iterate over `source` instead of retaining the list.

## Validation

With Node 20 or later:

```sh
node validate.mjs
```

The default discovers `cues-*.jsonl` beside the script. File and directory arguments are also accepted. It parses every non-empty line, validates each object against `schema.json`, reports the missing-or-null rate for every declared field, and exits non-zero if JSON parsing or schema validation fails. It also exits non-zero when it finds no cue shards.

The corruption check below copies one shipped shard to a temporary directory, changes the first row's `kind`, and runs the validator against that copy:

```sh
tmp_dir=$(mktemp -d)
cp cues-1919.jsonl "$tmp_dir/cues-corrupt.jsonl"
node -e 'const fs=require("fs"),p=process.argv[1],x=fs.readFileSync(p,"utf8").split("\n");const r=JSON.parse(x[0]);r.kind="not_a_cue";x[0]=JSON.stringify(r);fs.writeFileSync(p,x.join("\n"))' "$tmp_dir/cues-corrupt.jsonl"
node validate.mjs "$tmp_dir/cues-corrupt.jsonl"
```

The final command prints `FAIL`, identifies line 1, and exits with status 1. Removing the temporary directory is safe after inspection.

## Licence

The underlying parliamentary record is available under the [Open Parliament Licence v3.0](https://www.parliament.uk/site-information/copyright/open-parliament-licence/). Its required attribution string is exactly:

> Contains Parliamentary information licensed under the Open Parliament Licence v3.0.

The licence does not imply that Parliament endorses this dataset, its makers, its analysis, or any reuse. Credit mySociety ParlParse as the derived source for the XML acquisition path. The licence status of mySociety's transformation and packaging is `UNKNOWN` here because this release does not confirm a separate term for it.

## Citation

Suggested citation:

> Artificial Wasteland. (2026). *Hansard Reaction Notation Concordance* (Version 1.0.0) [Data set]. https://artwaste.land/data/hansard-reactions/

Also include the exact parliamentary attribution string above and your access date. For row-level citation, add the row's `url` for HTML or the resolved XML file URL plus `speech_id` and `colnum`.

## LIMITATIONS

This counts notation, not noise. It measures how often the frozen parser found reporter-written forms in the available record. It does not measure how much the House laughed, cheered, interrupted, or made any other sound.

Do not claim that preceding words caused a cue. The dataset shows adjacency in an edited record only. Do not call a cue funny or assume humour; parliamentary laughter can be hostile, tribal, or directed at someone vulnerable. Do not claim words were spoken exactly as printed; Hansard is an edited, substantially verbatim report. Do not treat any associated audio as historical sound; any audio must be labelled as a synthetic score. Do not invent quotations, and do not generate likenesses of real politicians from these records.

Coverage changes at the 1909 Official Report seam and again at the 1919 acquisition seam. The early data is a sample with known source-date faults. The later data is a source-collection census only when all listed files succeeded. OCR errors, parser false positives, parser false negatives, truncation, incomplete entity decoding, missing provenance, and unlogged HTML fetch failures remain in the release.