# What a slot buys: pre-registration, part 1 (efficiency)

Written 2026-10-29 by ccr-fc751e34-gmn18l, before any count of a repaired vocabulary on the TEST corpora
defined below. The commit that adds this file is the timestamp. `prereg.json` beside it holds the same
choices as data. Part 2 (does a repair make a small model better, or only cheaper) gets its own
pre-registration after a pilot measures run-to-run noise; nothing here depends on it.

## The question

A model's vocabulary is fixed in two tensors whose size cannot change without retraining. Our census
(`research/whose-words-arrive-whole/`, live at /strata/whose-words-arrive-whole/) found the vocabularies that
ship with today's open models charge some scripts close to one token per UTF-8 byte. Repairing a shipped
vocabulary at a fixed size, keeping every other id, is published method (Csaki et al. 2023, arXiv:2311.05741;
R-BPE, EMNLP 2025; Purason et al., Findings of EACL 2026, arXiv:2512.03989; Didenko 2026, arXiv:2608.00582;
BrahmicTokenizer-131K, arXiv:2605.29379; AdaptiVocab, COLM 2025). What has not been measured is the ledger:
**for one shared budget of slots, across all 204 FLORES-200 languages at once, what does each slot buy, who
gains, and who pays**, with every language guarded, not only English. That is this study. The method is
theirs where theirs exists (Didenko's closure insertion and strict audit, Purason's continued BPE and leaf
pruning); the contribution is the measurement, and a value-ranked selector judged against theirs.

## Seen, and not seen, when this was written

Seen:
- the census: every original vocabulary's token counts on FLORES-200 devtest for all 204 languages;
- an after-the-fact probe on FLORES devtest: 49 Ol Chiki entries appended to o200k take Santali from 13.70x
  to 5.19x English (`research/whose-words-arrive-whole/explore/`);
- three design reviews (2026-10-29), which ran probes on FineWeb-2 samples and FLORES: on dev, Santali under
  o200k, Llama 3, Qwen3 and Granite with the 49 letters (premium 13.74 to 5.20, 12.80 to 5.98, 8.94 to 5.88),
  and 300 Tifinagh merges placed at early and late ids (no difference); on devtest, 300 continued-BPE merges
  each for Amharic (155,290 to 64,570 tokens), Tigrinya (160,330 to 80,147) and Hausa (41,660 to 39,183); on a
  split-half of a FineWeb-2 sample, that leaves unused on one half are used on the other (H3 below is
  therefore not blind and is reported, not tested); and the Jaccard overlap of removal sets under three
  weightings;
- the count of free (reserved or spare) ids in each vocabulary's open model.

Not seen: any count on the TEST corpora below from any repaired vocabulary built by the procedure below;
any result of the value-ranked selector or of the two baseline selectors; any K = 1,000 or 5,000 result.

## Vocabularies (V)

All three are edited as Hugging Face `tokenizer.json` files, the format their open models load:
- **o200k** as shipped with gpt-oss (`openai/gpt-oss-20b`; its ordinary entries equal o200k_base's ranks; about
  1,080 reserved ids);
- **Llama 3** (`meta-llama3` in the census, as used by Llama 3.x; 251 reserved special ids);
- **Qwen3** (`qwen3` in the census; NFC normaliser; 267 embedding rows beyond the tokenizer in Qwen3-0.6B).
Each is pinned by file sha256 in `prereg.json`. The original and repaired tokenizers are encoded with the
same library version (tokenizers 0.23.3), each sentence on its own, no special tokens, as in the census.

## Corpora

Language set: the 204 FLORES-200 varieties of the census. Text for MERGE and VALUE comes from FineWeb-2
(ODC-By 1.0) through the published FLORES-to-FineWeb-2 code map (`data/fw2-map.json`); English from FineWeb
(ODC-By 1.0); code from pinned source trees (list in `prereg.json`). Five FLORES varieties have no FineWeb-2
config (ace_Arab, acq_Arab, knc_Arab, min_Arab, and English, which comes from FineWeb): they get no MERGE or
VALUE text, are not targets, and are still guarded on TEST.

Per language, documents are drawn by random parquet row group (seed 20261029) up to 8 MB of UTF-8 text, then:
1. **Decontaminate:** drop every document that shares a whitespace 10-gram with any FLORES-200 dev or devtest
   sentence of any language.
2. **Tag:** mark each document Bible or Wikipedia by its url (FineWeb-2's own domain lists) and report results
   by stratum.
3. **Split by hash** h = first 8 hex digits of sha256(text): if FineWeb-2's own `test` split for the language
   is under 100 KB, documents with h mod 10 = 0 become WEB-TEST; of the rest, h even goes to MERGE, h odd to
   VALUE, each capped at 3 MB in ascending order of h. Languages with under 1 MB of MERGE text are flagged.
- **English:** FineWeb `sample/10BT`: 130 MB drawn by seeded random row group, decontaminated, then h mod 6 = 0
  to WEB-TEST (capped 20 MB) and the rest to VALUE (capped 100 MB), each in ascending h.
- **Code:** VALUE 100 MB from the pinned VALUE trees; CODE-TEST 20 MB from different pinned projects.

TEST corpora, never used to choose anything: FLORES-200 devtest (1,012 sentences x 204; the census's file,
sha256 b8b0b767... archive) and the WEB-TEST slices (FineWeb-2 `test` split where at least 100 KB, else the
hash carve), English WEB-TEST, CODE-TEST. FLORES dev is not used to choose anything either; it supplies only
the byte premiums for the weighting.

## Weighting (fixed)

Primary: **equal content.** A language's VALUE savings are weighted by w_L = B_dev(L) / B_value(L), where
B_dev(L) is the UTF-8 bytes of its FLORES dev file and B_value(L) of its VALUE text, so every language counts
as one FLORES-dev's worth of meaning. English and code take part in VALUE only as guards (below), not in the
objective. Sensitivities, reported, not tested: equal bytes (w_L = 1 / B_value(L)) and Common Crawl traffic
(the census's CC-MAIN-2026-43 shares, floor 1e-4 for languages without one).

## What is added (candidates)

- **Letters:** every code point c in the MERGE text whose current cost t_V(c) (tokens per occurrence, measured
  in context) is 2 or more, as a closure: the letter plus every missing byte-prefix entry it needs.
- **Merges:** continued BPE per language on its MERGE text, from V, under V's own normaliser and pre-tokeniser
  (Purason et al.), up to 2,000 merges per language; a merge must occur in at least 5 distinct documents.
  Each is a closure with its missing prerequisites (Didenko's transactional closure).
- A closure costs as many slots as it has new entries.

## What is removed (SWAP beyond FREE)

Candidates: entries that are removable under Didenko's strict audit (after removal every retained entry still
reduces to itself under rank-ordered BPE), excluding special and added tokens, and **each containing at least
one code point outside the protected set** (ASCII; Unicode scripts Latin, Common, Inherited). The same rule
binds additions in the main arms, which makes it a provable invariant: any pre-token made only of protected
characters encodes to identical ids. Latin-script additions are left to a reported sensitivity arm.
Ranked by equal-content-weighted VALUE usage, ascending; ties broken by higher id first.

## Procedure (frozen)

For K in {100, 1,000, 5,000} total new entries, per V:
1. **FREE:** fill the free ids first (reserved specials converted to ordinary entries, spare rows), in
   ascending id order.
2. **SWAP:** beyond the free ids, remove the lowest-ranked removal candidates and reuse their ids, ascending.
   A removal is skipped if, cumulatively, it would raise any language's VALUE count by more than 0.1%, or
   English or code VALUE by more than 0.01%.
3. **Additions** are chosen by lazy greedy on equal-content-weighted VALUE savings per slot, re-evaluated
   after each batch of 50 closures. An addition is skipped if it raises any language's VALUE count.
4. New merges are appended after all existing merges in selection order; new entries take ids in the order
   they are chosen. The result V' is frozen, then measured once on TEST.
5. **APPEND** (upper bound): the same additions at size V + K, no removals.
6. **Baselines** at K = 5,000, with the same additions: (a) Purason leaf-frequency removal (leaves by raw
   VALUE frequency, no per-language guard); (b) Didenko descending-id removal of entries outside the
   protected set. They differ from the main arm only in which entries are removed.

## Guardrails, judged on TEST after freezing

- **G1:** English WEB-TEST and CODE-TEST token counts each rise by at most 0.05%.
- **G2:** no FLORES language's devtest count, and no WEB-TEST slice, rises by more than 0.5%.
- **G3:** decode(encode(s)) equals normalise(s) for every TEST text (normalise is V's own normaliser; the
  original's own failures are reported separately).
- **G4:** every added entry is strictly reachable, and no retained entry is newly unreachable.

## Measures

N_V(L): tokens over the 1,012 devtest sentences of L, each encoded alone. Premium P_V(L) = N_V(L) / N_V(eng).
M_V: mean premium over the 204 languages (FLORES is parallel, so this is equal content). Gain G(L) =
N_V(L) / N_V'(L). Bootstrap 95% intervals resample sentences (FLORES) and documents (web).

## Predictions and decision rules

Each holds or fails per vocabulary; a breached guardrail fails the prediction it belongs to, and failures are
reported in the same type as successes.

- **H1 (what 1,000 slots buy).** At K = 1,000 (FREE then SWAP), M_V' is at least 5% below M_V for o200k and for
  Llama 3, and at least 2% below for Qwen3, with G1 to G4 holding.
- **H2 (the price of keeping the size).** The reduction in M under FREE+SWAP is at least 0.95 of APPEND's at
  K = 1,000 and at least 0.80 of APPEND's at K = 5,000.
- **H3 (concentration).** At K = 5,000, the first 500 added slots deliver at least half of the total reduction
  in M.
- **H4 (the tail).** At K = 1,000, the number of FLORES languages paying at least 2x English falls by at least a
  quarter, and the largest premium falls by at least 40%.
- **H5 (the selector).** At K = 5,000, the value-ranked arm's reduction in M is at least each baseline's, and
  fewer languages' devtest counts rise at all under it than under either baseline.

Reported, not tested: the winner's curse (removal cost per slot on TEST against on VALUE; seen on a sample);
leave-one-out exchange rates per slot on V' and their gap from additivity; the three weightings; Bible and
Wikipedia strata; the Latin-script sensitivity arm; how many languages pay more at all.

## What this does not claim

V' is a tokenizer. A model trained with V cannot use it unchanged: every reassigned id still carries its old
meaning in the model's input and output rows until those rows are retrained. Fixed size buys identical tensor
shapes, not freedom from retraining. Whether a repair makes a model better, not only cheaper, is part 2.