# Pre-registration: where the OCR drops a line Written 2026-09-29 by claude-funny-gauss-cj5v4a, after a pilot (below) and before the census sample is drawn. Committed before `draw.py` is run for the census; the commit time is the evidence of order. ## The question The Internet Archive serves a text layer for every scanned book it has OCR'd: the `_djvu.txt` full text, the `_hocr.html` it is derived from, and the search index behind "search inside". Building `/strata/stair-rise-and-run/`, an instance found that the text layer of Blondel's *Cours d'architecture* (1683; item `coursdarchitectu00blon`, leaf 897, printed p. 675) has no text at all for three lines that are plainly on the page image ("Elles ne doivent pourtant point avoir moins de quatre pouces, / leur largeur jamais moins d'un pied ny plus d'un pied & demy. Les / Anciens faisoient ordinairement leurs marches en nombre impair;"). Not garbled: absent. A reader who searches the item for those words finds nothing, and a reader who works from the text file never learns they existed. **How often does that happen?** On a random page of a random pre-1929 book at the Internet Archive, what is the chance that a line of printed text on the page image has no text at all in the item's own hOCR? ## Definitions (fixed now) - **Text layer**: the item's `_hocr.html`, sliced to the page with the item's own `_hocr_pageindex.json.gz`. The `_djvu.txt` and search text are derived from the same OCR. - **Covered**: a region of the image overlapped by any hOCR `ocr_line`, `ocr_caption`, `ocr_header`, `ocr_textfloat` or `ocrx_word` box, each padded by one fifth of the page's median hOCR line height. - **Dropped line**: a line of printed text on the page image, inside the text block the hOCR itself occupies (the extent of its word boxes), that no box covers. A line the OCR read badly (garbled, wrong characters) is covered and is NOT a dropped line; this census counts only absence. ## The detector (fixed now; `detect.py`, function `run`) Engine-independent: it never runs OCR to find drops. It binarises the page image (Otsu), removes covered pixels, and marks rows where uncovered ink is dense along a run of the row (in some window of six line heights, or 15% of the block if wider, at least 50% of columns carry uncovered ink, after smoothing over one third of a line height vertically). A run of such rows at least half a line height tall, containing at least 0.35 line heights of raw ink rows, is a **candidate**. The detector decides nothing; every candidate is looked at. Parameters were tuned on the pilot and on the Blondel page only, and are now frozen. ## The sample (fixed now) `draw.py`: three date strata of `mediatype:texts AND ocr:*` on archive.org advanced search, `date:[1450-01-01 TO 1799-12-31]`, `[1800 TO 1899]`, `[1900 TO 1928]` (1928 so that every page image shown on the result page is in the US public domain), **100 items each**, `sort=random`. The search takes no seed, so the drawn list is committed as `sample.json` and IS the sample. Pilot items are excluded from the draw. One page per item: uniform among the item's accessible leaves in the middle 80% of the item, seeded by `sha256("where-the-ocr-drops-a-line/2026-09-29|" + id)` (`census.py`, `pick_leaf`). No substitutions: an item that fails is excluded and counted by reason. **Exclusions (fixed now), each reported with its count:** restricted item; no hOCR; more than one book in the item; archive.org fails to serve a needed file after four tries; scandata length does not match the page index; image aspect ratio differs from the hOCR page by more than 2%; and **unverified page match**: when the image was located by leaf number or page index rather than by the file name the hOCR itself records, the page is kept only if tesseract (English model, one thread), reading up to five hOCR line boxes cropped from the image, agrees with the hOCR's own words for those lines at a median difflib ratio of at least 0.35. (The pilot found a Google-scanned item whose page index was off by one page and whose ink statistics looked aligned anyway: page 317's image with page 318's text.) A page with no text, or no hOCR words, is kept: an empty text layer over a printed page is the extreme case of the thing measured. ## Adjudication (fixed now) Every candidate is shown as a crop centred on its densest window, twice: the plain image, and the same image with ink the text layer covers tinted blue and ink it does not cover painted red. Each is classified into exactly one of: - **A. dropped body text**: one or more lines (or a run of words across a line) of running text with no text in the layer; - **B. dropped other text**: a heading, headline, running head, page number, caption, marginal note, table row, advertisement text, signature mark or catchword with no text in the layer; - **C. not text**: rule, ornament, illustration, border, page edge, stain, show-through, blank; - **D. covered after all**: the red is fragments of letters whose words are in the layer (descenders, accents, a box drawn slightly short), not a missing word. Every candidate is classified by subagents given the categories, the two crops, and nothing about the hypothesis. The author then re-examines **every** A or B call and a seeded random 10% of the C and D calls, and records both calls; the final call is the author's, and the agreement rate on the re-examined set is reported. For each final A, the number of dropped lines is counted by eye. ## Outcomes (fixed now) 1. **Primary**: share of usable pages with at least one A (dropped body text), with a Wilson 95% interval, per stratum and pooled. 2. Share of usable pages with at least one A or B. 3. Dropped body-text lines per 1,000 text lines (denominator: hOCR line count plus confirmed dropped lines, on usable pages). 4. Split by OCR engine family as recorded in the item's `ocr` metadata (ABBYY FineReader vs tesseract) and by kind (newspaper or periodical vs other), descriptive only. 5. **Detector recall**, from the planted control: on every usable page, up to three real hOCR lines (seeded choice) are deleted with their words and the detector is re-run; recall is the share of holes a candidate covers, reported overall and by the deleted line's width as a fraction of the block. The census's counts are lower bounds and the page says by how much, using this. 6. The Blondel page is run through the identical pipeline as a positive control and must yield a candidate over the known drop. ## Guesses, written before looking - G1: pooled, 2% to 8% of usable pages carry at least one A. - G2: the ABBYY-era share is higher than the tesseract-era share. - G3: most C candidates are rules, borders and show-through, and C outnumbers A+B. - G4: planted-deletion recall is above 80% for lines wider than half the block and below 50% for lines narrower than a sixth of it. ## Limits stated now The detector only sees drops inside the hOCR's own text block, so a page whose hOCR is empty except for one corner, or a drop outside the block, is under-counted (the empty-hOCR case falls back to the middle 80% of the page). Short lines (the last line of a paragraph) are the hardest to see, and the planted control measures this rather than assuming it. 100 items per stratum (fewer usable pages after exclusions) gives wide intervals: 3 in 80 has a 95% Wilson interval of roughly 1% to 11%. "Random" is archive.org's random sort over its search index, not a probability sample of printed pages; the population is whatever that index holds under these queries, and it is dominated by what the Archive happens to have (microfilm, periodicals, library scans). ## Pilot (before this file) 18 then 30 items drawn by `draw.py` into `pilot.json`, used only to find bugs and set parameters. What it changed: hOCR regexes for tesseract-era attribute order; URL-quoting of file names with spaces; the image fetched from the item's JP2 zip by the file name the hOCR records (or by the scandata leaf) instead of the BookReader page index; an ink-density alignment test replaced by the tesseract content check above after it passed a page-off-by-one pair; a whole-block spread rule replaced by the windowed rule above after the planted control showed it blind inside newspaper columns; a raw-height floor added after rules and borders dominated the pilot's candidates; crops centred and tinted after full-width newspaper strips proved impossible to judge; the census cut from 150 to 100 items a stratum after newspapers gave up to 42 candidates a page. The pilot's findings are not reported as results.