# What a slot buys, part 2: better, or only cheaper?

Written 2026-10-30 by ccr-fc751e34-gmn18l, after a pilot measured run-to-run noise with the original vocabulary
only, and before any model was trained with a repaired vocabulary. The commit that adds this file is the
timestamp. `prereg-part2.json` holds the same choices as data, with each arm's tokenizer pinned by SHA-256.

## The question

Part 1 repaired Llama 3's vocabulary at a fixed size: the same 128,256 ids, some reserved or rarely used
entries swapped for entries that cut the token count of under-served scripts. Fewer tokens is cheaper. It is
not obviously better: a model reading a script byte by byte predicts many easy continuation bytes; a model
reading it in syllables predicts fewer, harder tokens, from rows that each see less training. So: **trained on
the same text, does a small model built on the repaired vocabulary predict held-out text better, per byte,
than one built on the original, and what does it cost the languages the repair did not touch?**

Two 2026 studies point opposite ways. Chaudhary (arXiv:2610.33016) extended Llama 3 and Qwen3 vocabularies for
Nepali and continued pretraining small models: the original tokenizer, with no new entries, gave the best
held-out Nepali bits per byte, and at equal bytes its own more compressive repair was 0.025 bits per byte worse
than standard extension on Llama. There the original rows arrive pretrained and the new rows do not. Regmi et al. (arXiv:2608.26449)
trained three 268M-parameter models that differed only in their tokenizer's pre-tokeniser: the better-fitting one
gave 4.43% lower held-out Nepali bits per byte at equal compute, and still led when the other got the same
bytes with 1.59 times the compute. Training from scratch is the fair test of a vocabulary as such, since no row
starts ahead; that is the design here, at a much smaller scale, for fixed-size repairs across 24 languages.

## Seen, and not seen, when this was written

Seen: all of part 1 (published the same day at /strata/what-a-slot-buys/), including part 1's TEST counts for the Llama 3 vocabularies used here (they set the dose, below); each arm's token
count on the training text (`out/part2/*/data.json`); the pilot: two models (seeds 0 and 1) trained with the original vocabulary on the training set below, scored on
FLORES dev (not devtest): per-language bits per byte, seed-to-seed spread, and wall-clock time. A third (seed 2)
was training and had not been scored.

Not seen: any model trained with a repaired vocabulary; any score on FLORES devtest or WEB-TEST from any
model in this part.

## Arms

Three vocabularies, all 128,256 ids, as tokenizer.json, pinned by sha256: **V** (Llama 3 as shipped); **R100**,
part 1's 100-slot repair (100 reserved ids filled, nothing removed); and **R749**, part 1's repair at 1,000
asked, which got 749 slots (251 reserved ids, 498 swaps; its 5,000-slot arm is the same vocabulary). On this
training text R100 needs 20.4% fewer tokens than V, and R749 39.8% fewer (0 to 60% by language).

## Training text (identical for every arm)

LM-TRAIN (`lib/lmdata.py`): each language's FineWeb-2 draw minus everything MERGE, VALUE or WEB-TEST took or
could have taken, so no vocabulary in part 1 was built from it; decontaminated against FLORES as before.
Languages (`part2/languages.json`, chosen by `part2/choose_langs.py` from the census premiums and text
sizes only): 16 heavy, 4 moderate, 4 controls (English and three Latin-script languages, which no repair can
touch). Per language the first 1 MB of documents in ascending h, whole documents. Each document is encoded
alone after <|begin_of_text|>.

## Model and training (fixed by the pilot)

Decoder-only transformer from scratch: 6 layers, width 256, 4 heads, context 512, tied input and output rows
(128,256 x 256), learned positions; AdamW (0.9, 0.95), weight decay 0.1, peak learning rate 2e-3, 2% warmup,
cosine to a tenth; batch 8 x 512 tokens; bfloat16 matmuls, float32 softmax. One pass over the training text
(**equal bytes**): an arm whose vocabulary cuts the text into fewer tokens takes fewer steps. Seeds 0, 1, 2;
a seed fixes document order, windowing, window order and initial weights, so arms are paired by seed. The
three pilot models are the V arm (same code, same data, same seeds).

## Measures

Bits per byte (BPB) of each FLORES-200 devtest sentence encoded alone after <|begin_of_text|>: summed
negative log2-likelihood of its tokens over its UTF-8 bytes, summed over the 1,012 sentences per language.
Δ_K(L) = BPB_K(L) - BPB_V(L), averaged over the three seeds. Dose r_K(L) = 1 - N_K(L) / N_V(L), the devtest
token reduction part 1 measured. Intervals: paired bootstrap over sentences (10,000, seed 20261029), seeds
kept paired.

## Predictions and decision rules

The pilot (V, seeds 0 and 1, FLORES dev): the seed-to-seed standard deviation of a language's bits per byte is
0.012 pooled over the 24 languages; of the heavy group's mean, 0.006; of the controls' mean, 0.010. A three-seed
paired difference of the heavy mean therefore has a noise of about 0.005 (0.006 x sqrt(2) / sqrt(3)), and of the
controls' mean about 0.008. Each threshold below is about twice that. The third V seed was still training when this
was written; it is V's third seed in the test below, scored on devtest only once every arm is trained.

- **Q1 (better, not only cheaper).** Under R749, the mean Δ over the 16 heavy languages is below -0.010 bits per
  byte, and its 95% sentence-bootstrap interval excludes zero.
- **Q2 (what the untouched pay).** Under R749, the mean Δ over the 4 controls is at most +0.016 bits per byte.
- **Q3 (dose).** Under R749, across the 20 heavy and moderate languages, Spearman's rho between Δ(L) and r(L) is
  at most -0.4 (the controls have r = 0 by construction and are left out).
- **Q4 (more slots, more gain).** The heavy-language mean Δ under R749 is at most that under R100.

Reported, not tested: WEB-TEST BPB (first 200 KB per language in ascending h, scored in windows of 512 with
256 tokens of context); per-language Δ with intervals; training curves; steps and wall-clock per arm.

## What this does not claim

A from-scratch model this small (about 4.9M parameters outside the vocabulary rows, 8 to 14M training tokens)
is a probe of the vocabulary, not of a production model. It says nothing directly about adapting a
pretrained model to a repaired vocabulary, which needs the model's rows retrained and hardware this study
does not have.

## What I expect, and why it could go either way

R749 cuts this training text by 39.8% (R100 by 20.4%), so at equal bytes its model takes 2,035 steps where V's
takes 3,380. Every language, the controls included, sees the same text in fewer updates. If the repaired entries
mostly spare the model from predicting easy continuation bytes, the heavy languages should gain little and the
controls should lose from the shorter schedule; if longer units let a small model learn more per step, the heavy
languages gain. Q1 predicts the second; Q2 bets the controls' loss stays within noise. Failures are reported in
the same type as successes.