# The three audits, in the order they happened

Each round: 150 ghazals drawn at random (seeded; ids in `round-N/ids.json`) from `data/ghazals.tsv`,
never reused across rounds. Only the rhyme lines were shown (`round-N/audit.md`). Two readers per
round, each a separate instance of the same model that built the page (Claude), each told to read
by hand, to run no code, and to open no other file. Neither saw the detector's answers. The rule they
were given is `round-N/RULE.md`. Their answers are `reader-*.txt`, unedited.

Scores are "the reader's answer agrees with the rule" out of 150, for whether a radif exists; and, for
the text, the number of ghazals where the rule and both readers found one and all three give the same
words. Re-run any round
with the rule as it stands now: `python3 research/the-word-that-comes-back/audits/score.py
research/the-word-that-comes-back/audits/round-N/ids.json research/the-word-that-comes-back/audits/round-N/reader-*.txt`.
The live result is in `score-now.json`.

## Round 1, a development set (it changed the rule)

The rule was fixed before the draw (v1). The two readers agreed with each other on 150 of 150.

| rule | readers A and B |
| --- | --- |
| C, the page's rule | 148 of 150 on presence |
| C, the text | 106 of 108 |
| S, as the edition spaces words | 143 of 150 |

What the four disagreements showed:

- **G049 and G122: a real fault.** The lexical split cut words that the edition never wrote apart anywhere in the poem. In G049 باشد was read as با + شد. In G122 بیدارش کند gave a false شکند.
  - Fix (v2): a refrain must be written as a separate word in at least one of its lines.
- **G057: a real fault.** One line wrote the rhyme vowel as آ (درآ), so the shared rhyme letter was missed.
  - Fix (v2): decompose (NFD) before stripping marks, so the madda goes the way of the other vocalisation marks.
- **G047: not a fault.** A loose rhyme in one line. The strict rule refuses it by design, and the tolerant rule (D) accepts it.

Two consequences of v2 appeared on re-scoring. The displayed refrain lost its madda (امد for آمد), which was cosmetic, fixed in v3 by displaying the edition's own letters. The madda-stripping also let یاد آری read as یا + داری.

- Fix (v3): a hidden boundary needs a preverb in front (بر، در، فرو، فرا، وا، باز، ور), or else at least three letters that are a corpus word.
- Round 1 under v3: 149 of 150, text 108 of 108. G047 is the one left.

## Round 2, fresh (and it found a bug of mine)

Drawn after v3 was fixed, from ghazals not in round 1. The readers agreed with each other on 149 of 150. In G024 they split on whether a joined ‑ست is the verb.

| rule | reader C | reader D |
| --- | --- | --- |
| C, presence | 146 | 147 |
| C, text (rule and both readers the same) | 104 of 105 | |

**G018 and G130: a bug introduced in v2.** With the madda stripped, آید ("comes") became اید, which is on the list of bound suffixes (-īd). آن would have become ان in the same way.

- Fix (v4): the bound-suffix test reads the edition's own spelling.

**G133: a cost of v3.** سرگرفتی hides گرفتی behind سر, which v3's three-letter rule refused.

- Fix (v4): add سر to the list. The list was chosen by a count over the whole corpus, not from this ghazal. Of the two-letter heads that hide a refrain boundary, بر (1,030 ghazals), در (433) and سر (360) are the three that are real verbal heads. The next ones down (خو, بی, خر) are chance letter overlaps. Reproduce it with `node research/the-word-that-comes-back/preverbs.mjs <dir>`.

**G025: not fixed.** The edition joins تو and می (تومی کنند). The rule finds کنند where the readers find تو می‌کنند.

Round 2 under v4: 149 of 150 (reader C) and 150 of 150 (reader D), text 107 of 108.

## Round 3, fresh, with the rule frozen: the number the page reports

Drawn after v4, from ghazals in neither earlier round. Nothing was changed after it. The readers agreed with each other on 150 of 150.

| rule | readers E and F |
| --- | --- |
| C, the page's rule, presence | **148 of 150** |
| C, the text (rule and both readers give the same words) | **104 of 105** |
| D, the tolerant rule | 146 of 150 |
| S, as the edition spaces words | 141 of 150 |

The misses, left as they are:

- **G051.** The readers took کردن as a refrain behind a joined head. The rule did not.
- **G141.** One line spells ماجرا as ماجری. The strict rule refuses it; the tolerant rule accepts it.
- **G022.** The refrain is ز من to the readers and من to the rule, which never lets a one-letter word other than و begin a refrain.

## What this does and does not show

It shows that the rule does what its written definition says, on ghazals it was not tuned on, about 98 or 99 times in 100.

It does not show that the definition is the right one, and three things limit it:

- A scholar of Persian prosody might draw the line elsewhere. The attached ‑ست, and suffixes carried by the refrain, are the live disagreements.
- The readers are not independent of the builder. They are the same model, told to work without code.
- They are not Persian scholars.