# The Accent of Nowhere: the method, exactly

This is the specification the lab code (`engine.mjs`, `lib.mjs`, `analyze.mjs`, `m6.mjs`,
`predictions.mjs`) implements and the verifier (`verify-the-accent-of-nowhere.mjs`, at the
repository root and published at `/checks/`) implements again by its own arithmetic. Where
this file and the code disagree, one of them is wrong. `PREREGISTRATION.md` fixed the measures
before any tongue was scored; `DEVIATIONS.md` lists the eight departures (D1 to D8).

## 1. Words

`normalize(text)` (N): Unicode NFD; delete every code point of category Mn; lowercase; replace
ß with ss, æ with ae, œ with oe, ø with o, đ and ð with d, þ with th, ł with l, ı with i, ŋ
with ng, ħ with h, ĸ with k, ſ with s, and each of ʻ ’ ‘ ʼ ′ ` with the apostrophe `'`; split
on every run of characters that are not `a` to `z` or `'`; strip apostrophes from both ends of
each piece; drop empty pieces. The result is the list of words (tokens), in order.

`types` are the distinct words, sorted (JavaScript's default string sort, which is by UTF-16
code unit, so `'` sorts before every letter).

The alphabet is the 27 characters `abcdefghijklmnopqrstuvwxyz'` in that order (LETTERS).

## 2. Tables and the model

A table is the multiset of padded trigrams of a list of words: each word w is padded to
`^^w$`, and every position i from 2 on contributes the three characters `p[i-2] p[i-1] p[i]`.
So each letter of w, and one `$`, is "predicted" once, by a key whose third character it is.
`n` is the total count. Packed form: keys sorted, each followed by its count in base 36, joined
by single spaces.

The model of a table is Witten-Bell interpolation, order 3 down to uniform over 28 symbols
(the 27 letters and `$`). With c3[abc] the table's counts:

- N = sum of all counts; c1[c] = sum over ab of c3[abc]; T0 = number of distinct c with c1[c] > 0.
- c2[bc] = sum over a of c3[abc]; ctx1[b] = sum over c of c2[bc]; t1[b] = number of distinct c with c2[bc] > 0.
- ctx2[ab] = sum over c of c3[abc]; t2[ab] = number of distinct c with c3[abc] > 0.
- p1(c) = (c1[c] + T0/28) / (N + T0), or 1/28 if the table is empty.
- p2(c | b) = (c2[bc] + t1[b]·p1(c)) / (ctx1[b] + t1[b]), or p1(c) if ctx1[b] = 0.
- p3(c | ab) = (c3[abc] + t2[ab]·p2(c | b)) / (ctx2[ab] + t2[ab]), or p2(c | b) if ctx2[ab] = 0.

`H(model, words)` = minus the sum of log2 p3 over every predicted character of every word,
divided by the number of predicted characters (letters plus one `$` per word). Bits per
character. Words are scored as types unless said otherwise: each distinct word once.

Each language's types were split once: sort, Fisher-Yates shuffle with the generator of §7
seeded by `seedOf(code)`, first 80% (rounded) is the model part, the rest is held out.
`data.json` ships the packed model-part table of each of the 38 languages whose licence
allows it (`panel[].table`).

## 3. Rulers, distance, rank (M2, M3)

For a language h taken as the ear, `panel[h].floor` = H(model_h, held-out types of h), and
`panel[h].ruler[i]` = H(model_h, held-out types of atlas language i) for the 730 atlas
languages in the order of `data.atlas` (`null` where the atlas language is h itself).
`vals` = the non-null ruler values of atlas languages whose code is not h, sorted ascending
(729 values when h is in the atlas, 730 when it is not). `median` = quantile(vals, 0.5).

`quantile(sorted, q)`: pos = (len-1)·q; linear interpolation between sorted[floor(pos)] and
sorted[ceil(pos)].

For a tongue X with types and home h: Hx = H(model_h, types of X);
**E = (Hx - floor) / (median - floor)**; **rank = number of vals strictly below Hx**; `of` =
len(vals). The second home, where one is registered, is scored the same way.

## 4. Nearest (M1) and the bootstrap

Nearest = the language of the pocket panel (46 languages, order `data.pocketOrder`) with the
smallest H(model_L, types of X). Eight pocket languages' tables are not shipped (no-derivatives
licences), so outside the repository only the 38 shipped ones can be recomputed;
`tongues[].expected.top` carries the lab's five smallest H values over all 46. Inside the
repository `research/the-accent-of-nowhere/panel/pocket.json` holds all 46 (`train` packed).

Bootstrap: B = 2000. `rand = rng(seedOf(id + '|boot'))`. For each b: draw n indices
`floor(rand()·n)` (n = number of types, indices into the sorted types), in order. The
resample's H under a language is (sum of the picked words' bits) / (sum of the picked words'
predicted characters). The winner of resample b is the language with the smallest H (first in
pocket order on a tie). `boot` = each language's share of wins, largest first (the lab keeps the five largest). "Called" means
the largest share is at least 0.8. E for each resample uses the home's H; **Elo, Ehi** =
quantile(sorted resample Es, 0.025 and 0.975). The random stream is consumed only by the index
draws, so E's interval can be recomputed from the home table alone.

The `id` is the tongue's id in `data.tongues`, or the cell's id in `data.cells`
(`cell-<arm>-<who>-<prompt>`).

## 5. Letters (M4)

Tongue counts: occurrences of each of the 27 characters over the tongue's types. Home counts:
the same over all of the home's types (`homes.json`, which lists them).

- Registered: f_t(c) = (count_t(c) + 0.5) / sum over all 27 of (count_t + 0.5), and the same
  for home; log2ratio(c) = log2(f_t(c) / f_h(c)).
- Matched totals (D5): first scale every home count by (total tongue letters / total home
  letters), then proceed as above.

In each version: `top3` = the three characters with the largest log2ratio (ties in LETTERS
order); `z` = log2ratio of z; `rho` = Spearman's rank correlation (average ranks for ties)
between the 27 log2ratios and the 27 values log2 f_h(c) of that version. The page prints
"times its share" as 2^log2ratio (rounded to an integer at 10 or more, else one decimal).

`meanLen` = mean letters per type. `vowelFinal` = share of types whose last character is one
of a e i o u y. The home's own values are over `homes.json`.

## 6. Structure (M5) and the noise control (C3)

`selfCV(words, seed)`: s = Fisher-Yates shuffle of words with rng(seed); fold f (0 to 9) tests
the words at positions i with i mod 10 = f against a model trained on the others; the result
is total bits over total predicted characters across the ten folds.

`shuffleLetters(words, seed)`: one generator rng(seed); each word in order is replaced by the
Fisher-Yates shuffle of its characters.

For a tongue with at least 20 types (in sorted order), with id:
- cv = selfCV(types, seedOf(id + '|cv'))
- sh = selfCV(shuffleLetters(types, seedOf(id + '|sh')), seedOf(id + '|cv'))
- **orderGain = sh - cv**
- home samples: all = the home's types, sorted; rs = rng(seedOf(id + '|home')); for s = 0 to
  199: sample = the first n of shuffle(all, rs) (rs carries on from sample to sample);
  hs[s] = selfCV(sample, seedOf(id + '|hcv|' + s)); for s below 40 also the home's order gain
  = selfCV(shuffleLetters(sample, seedOf(id + '|hsh|' + s)), seedOf(id + '|hcv|' + s)) - hs[s].
  **homeMedian** = median of hs; `homeShareBelow` = share of hs below cv; **homeOrderGain** =
  median of the 40.

C3: r = rng(seedOf(id + '|c3')); each type, in sorted order, is replaced by a string of the
same length whose letters are LETTERS[floor(r()·26)] (a to z, no apostrophe). The strings are
scored as they come, one for each type, duplicates kept (two types of one length can draw the
same string; the list is not reduced to distinct strings): nearest ear, and E on the home's ruler.

## 7. The seeded generator

`seedOf(str)`: FNV-1a, 32 bits: h = 0x811c9dc5; for each UTF-16 code unit: h ^= unit;
h = (h × 0x01000193) mod 2^32.

`rng(seed)`: mulberry32. State a = seed. Each call: a = (a + 0x6D2B79F5) mod 2^32; t = a;
t = imul(t ^ (t >>> 15), t | 1); t ^= t + imul(t ^ (t >>> 7), t | 61); return
((t ^ (t >>> 14)) >>> 0) / 4294967296.

Fisher-Yates: for i from len-1 down to 1: j = floor(rand()·(i+1)); swap i and j.

## 8. The model arm

A reply is parsed as: the text between the first pair of triple-backtick fences (an opening
fence may carry one of the words text, txt, plaintext, plain, none, which is dropped; anything
else on the opening line is content), or the whole reply if there is no such pair. An opening
fence that is never closed is not a pair: the whole reply is scored (six replies are like
that, and none of them has a word on the fence's line). Then N.
A cell pools the words of its draws (same arm, same model or setting, same prompt) and is
scored as one tongue with home English.

## 9. Word for word (M6)

Over `data.martianLines` (each a Martian line and the French line Flournoy prints under it):
registered = share of lines where N(tongue) and N(gloss) have the same number of words;
"labels removed" = the same after deleting from the tongue side every `[A]`-style label
(`\[[A-Z]\]`), every parenthesis with its contents, and any remaining square brackets.

The comparison registered with it (H8) needs the corpus: for each other pocket language, the
share of New Testament verses in which N(the French verse) and N(that language's verse) have
the same number of words. The largest of those shares is the reference.

Exploratory, not registered: over the lines whose cleaned count equals the gloss's, pair the
words by position and keep each distinct pair once. `r` is Pearson's correlation between the
two words' lengths in characters, `sameLength` the share of pairs of equal length. The chance
values come from 10,000 shuffles of the French side against the Martian side (Fisher-Yates
with rng(seedOf('m6|perm')), one generator for all the shuffles): the mean `sameLength`, and
the largest `r` any shuffle reached.

## 10. The predictions

As worded in `PREREGISTRATION.md`, over `data.tongues` entries with `counted` true and at
least 100 types where a prediction speaks of "the tongues with at least 100 types". H3: classes
revealed, ecstatic, artistic, ids not starting with `enochian`, all E below 1. H4 and H5 are
evaluated under the registered letter formula (and reported under both). H6: median E of
entries with `apriori` true against median E of counted revealed and ecstatic tongues with at
least 100 types. H7: counted revealed and ecstatic tongues with at least 20 types (those that
have M5), share with cv below homeMedian, against 0.75. H9 to H11: the four Claude cells of
each prompt. Outcomes: held, failed, held in part (one of two clauses), untestable.

## 11. What the page prints

Every figure is `<span data-v="KEY">VALUE</span>`, computed from values kept to nine decimals
and rounded once, here. `results/figures.json` holds every key with its value as built.

For a tongue the prefix is `t.<id>`; for a cell it is `c.<setting or short model name>.<prompt>`
(`data.cells[].key` gives it). After the prefix:

- `types`, `tokens`; `E`, `Elo`, `Ehi` (two decimals); `rank`, `of`; `E2`, `rank2` (the second home).
- `near`: the name of the registered nearest ear of the 46. `nearshare`: that ear's share of the
  2,000 resamples, as a percentage rounded to a whole number.
- The letters, all in the matched-totals version (D5), never the registered one: `top3` (three
  characters separated by spaces), `ztimes` (2^log2ratio of z: a whole number at 10 or more,
  else one decimal), `zcount` (how many z's in the types), `rho` (two decimals). `zwords`: how
  many types contain a z.
- `vf`, `homevf` (percent, rounded); `len` (one decimal).
- Structure: `gain` (orderGain), `cv` (the tongue's own ten-fold figure), `homecv` (homeMedian),
  `homegain` (homeOrderGain), two decimals. The lab also keeps `homeLo` and `homeHi`, the 0.025
  and 0.975 quantiles of the 200 home samples; the page does not print them.
- `noiseE`: E of the noise control (two decimals).
- `on.<code>.E`, `.rank`, `.of`: the tongue on the ruler of pocket language `<code>`, as in section 3.
- `repE`, `reprank`, `repof`, `repnear`: the replication on the UDHR (not recomputable from
  what the page ships).

Other keys:

- `p.<H id>`: the outcome of a prediction.
- `h4.n`, `h4.neg`, `h5.n`, `h5.z`, `h5.exotic`: over the counted tongues with at least 100
  types, under the REGISTERED letter formula: how many there are, how many have rho below 0,
  how many have z's log2ratio above 0, how many have one of z x k q in their top three.
  `h4.negm`, `h5.zm`, `h5.exoticm`: the same three counts under the matched-totals formula.
- `x.A.minbestrank`: over the four Claude prompt-A cells, each cell's smallest rank on any of
  the 46 pocket rulers, then the smallest of the four. `x.B.gainmin`, `x.B.gainmax`: the least
  and greatest orderGain among the four Claude prompt-B cells. `x.BA.min`, `x.BA.max`: the least
  and greatest of (E of the prompt-B cell) / (E of the prompt-A cell) over the four settings,
  one decimal. `x.cells.claude`: the number of Claude cells; `x.apos.claude`: how many of them
  have the apostrophe in their top three.
- `c3.n`: how many tongues and cells have a noise control; `c3.three`: in how many of those the
  noise is heard best by Vietnamese, English or Tok Pisin.
- `ex.deu.herrlichkeit`, `ex.deu.mglwnafh`: bits per predicted character of those two words
  under the German model, one decimal.
- `quc.apos`, `eng.apos`: the share of apostrophes among the predicted characters other than
  `$` of the K'iche' and the English model tables, as a percentage.
- `m6.lines`, `m6.equal`, `m6.share`; `m6.cleanequal`, `m6.cleanshare`; `m6.refname`,
  `m6.refshare` (the reference of section 9); `m6.pairs`, `m6.r`, `m6.samelen`,
  `m6.samelennull`, `m6.rnullmax` (the exploratory count of section 9).
- `c1.a<alpha>.n<n>` and `s0.right`, `s0.of`: the controls (`controls.mjs`).
- `nb.<code>.ear`, `.E`, `.rank`: section 12.
- `atlas.n`, `pocket.n`, `pocket.shipped`, `rep.n`; `draws.claude`, `draws.rail` (draws scored),
  `draws.railasked`, `draws.railempty`, `rail.listed`, `rail.admitted`, `rail.silent` (from the
  rail's own log, `model-arm/rail/results.json`: draws asked, draws that came back empty, model
  ids the rail lists, ids admitted, ids that were probed and did not answer).
- `ocr.martian.total`, `ocr.martian.inwords`: counts from the transcription notes of the
  Martian file.

## 12. Real languages on each other's rulers

For a pocket language X and each other pocket language L: `vals` = L's ruler over the atlas
languages other than L and other than X (X's own entry in the atlas is left out, so that a
language is not compared with itself), sorted; h = H(model_L, held-out types of X); rank = the
number of vals strictly below h; E = (h - floor_L) / (quantile(vals, 0.5) - floor_L). X's
nearest ear is the L with the smallest rank, and of two with the same rank the one with the
smaller E. `nb.<X>.ear` is its name, `nb.<X>.E` its E (two decimals), `nb.<X>.rank` its rank.

Esperanto is one of the atlas languages (code `epo`; DEVIATIONS.md, D8), and the page places
it the same way, with X = `epo` and L each pocket language in turn: `epo.on.<L>.E` (two
decimals), `epo.on.<L>.rank`, `epo.on.<L>.of` (the number of vals), and `epo.near1`,
`epo.near2`, the names of the two ears with the smallest rank (ties to the smaller E). Here h
is L's ruler value for `epo` as shipped, since the held-out words themselves are not shipped.
