The Artificial Wasteland · how a language model reads

What a Slot Buys

A language model reads numbered pieces from a list fixed before it was trained, and the lists that ship charge some languages ten times what they charge English for the same sentence. In three open models’ vocabularies we respent the slots nobody used, and entries real text barely needs, on the pieces those languages lack, without moving any other entry. gpt-oss’s 1,000 unused ids cut the mean cost against English by 11% and made no language pay a token more; Qwen3’s 5,000 slots cut it by 16%; Llama 3 had room for only 749, which cut it by 18% and cost Kashmiri 2%. Eight of fifteen predictions written down first held.

A language model never reads letters. It reads numbers, one for each piece of text, from a list fixed before it was trained. Llama 3’s list has 128,256 entries. Change one entry and the number the model learned for it means something else, so the list a model ships with is the list it keeps.

But not every entry is spoken for. Some ids are reserved and never filled; gpt-oss’s model even has 1,069 rows its tokenizer never uses. Some entries hold scraps of spam. Meanwhile the census before this page found the same vocabularies charging Santali, Shan, Tibetan and Tifinagh-script Tamazight ten times or more what they charge English for the same sentence. This page respends the spare and the unneeded on the pieces those languages lack, keeping every other entry at its number, and measures what each slot buys and who pays.

1 · the same sentence, twiceOne sentence, before and after

The census showed one sentence in every language: Thus, the pencil was a good friend to many people when it came out. Here it is again in the languages the repair served most, and in a few it did not touch, cut by each vocabulary as shipped and then after its repair at the 1,000 slots asked for (749 for Llama 3, which had no more to give). Every box is one token, one number the model receives. A box of bytes is part of a letter: the vocabulary holds no single entry for that whole character, so the model is handed its UTF-8 bytes in pieces.

Look at the repaired Santali line. Many new tokens hold one whole letter and the first two bytes of the next. Every Ol Chiki letter begins with the same two bytes, so a merge learned from text glues them on. Byte-pair encoding learns what text repeats; it does not know where a letter ends.

2 · every languageTwo hundred and four languages, before and after

Each dot is a language. Across is what it costs against English with the vocabulary as shipped; down is what it costs after the repair. A dot on the dashed diagonal did not change, below it got cheaper, above it got dearer. Every language with web text to its name was guarded, 199 of the 204 and English: a repair may remove an entry only if no language’s text costs more than 0.1% for it, and English and code more than 0.01%, measured on text set aside for choosing (web text for the languages, open-source code for code). Four languages have no text in the FineWeb-2 web corpus the guards came from and were only checked afterwards.

gpt-oss’s 1,000-slot repair needed no removals at all, because its model has 1,080 ids nobody uses: 11 reserved tokens and 1,069 rows beyond its tokenizer. Its mean cost against English fell from 2.12 to 1.88 times, 11.22% lower, and not one of the 204 languages paid a single token more. Santali went from 13.70 times English to 4.01. Llama 3’s repair fell further, from 2.99 to 2.45 times, 17.90% lower, because it started further behind: Shan from 14.79 times to 5.52, Tibetan from 11.19 to 4.63. But Llama 3 had only 251 empty ids, so the rest came from removals, and 31 languages ended up paying a little more. Qwen3 had 267 empty rows and found 733 removals that cost almost nothing: at 1,000 slots its mean cost fell from 2.74 to 2.43 times English, 11.27% lower, and the only text that rose at all was Korean web text, by 0.0007%. Shan went from 10.43 times English to 5.44, Tibetan from 7.23 to 3.60.

3 · what each slot buysThe first hundred are worth the most

Llama 3’s first 100 slots, all of them reserved ids, cut its mean cost 9.05% and made nothing dearer. The next 649 added less than nine points more. gpt-oss’s first 100 cut 6.85%, and its 1,000, every one an unused id, 11.22%. Qwen3’s first 100 cut 4.91%, its 1,000 11.27%, its 5,000 15.57%.

Only one of the three could free the 5,000 slots the study asked for. Under the guard, Llama 3 could give up only 498 of its 6,080 removable entries and reached 749 slots; gpt-oss gave up 1,390 of 9,445 and reached 2,470; Qwen3 found all 4,733 it needed and reached the full 5,000. What it gave up is different in kind: rare single characters (Kangxi radicals, rare Hangul syllables, Canadian syllabics, even an Egyptian hieroglyph), and 229 characters that Qwen3’s own text normaliser rewrites before the vocabulary ever sees them, 194 of them compatibility forms of Chinese characters, so they could never be used at all. Llama 3 and gpt-oss had little such deadwood: almost every entry outside the Latin alphabet earns its keep in some language. And for gpt-oss the swaps bought little: adding 1,390 swaps to its unused ids bought one point more, 12.27% against 11.22%.

4 · what was tradedOut went the spam; in came the letters

The entries the guard let go were the ones real text in 204 languages almost never produces. For gpt-oss about half of the 1,390 were Chinese, many of them gambling and pornography-site phrases (“lottery platform account opening”, “lottery forum”, “win every day”), with web furniture, place names and programming words beside them, and nearly half of the rest were Cyrillic. In May 2024 MIT Technology Review reported that of the 100 longest Chinese entries in GPT-4o’s vocabulary, which gpt-oss shares, all but three belonged to gambling or pornography. For Llama 3 a quarter of the 498 were Chinese, among them web boilerplate (“Chinese subtitles”, “page archive backup”), and most of the rest were Cyrillic word pieces, fragments of Thai and Korean, and Greek. In went letters and parts of letters for the scripts the census found paying most, then syllables and common word pieces.

5 · who paidThe guard watched the wrong text

The guard was checked on text it never saw: FLORES-200’s test sentences and a separate slice of web text for each language that has one. The plan allowed any one of them to rise by at most 0.5%. Under Llama 3’s repair, Kashmiri written in Devanagari rose 2.25%, all of it from 15 Devanagari entries the guard let go, among them the letter ज़ itself. Its guard text, from the FineWeb-2 web corpus, is a third Latin-script text, and much of the rest is broken extraction with vowel signs missing; it holds letters with the nukta dot (ज़, ड़) about a ninth as often as real Kashmiri does, so on it those removals cost Kashmiri 7 tokens. A guard is only as good as the text behind it.

For Llama 3’s removals, the cost on FLORES ran 2.6 times what the choosing text had shown, while on the web slices it was about the same (1.04 times): web text predicts web text. Under gpt-oss’s largest repair, Yiddish rose 0.64%, also past the limit. Qwen3’s swaps, rare characters rather than words, broke nothing: at 5,000 slots the worst rise anywhere was Korean web text, by 0.03%.

6 · whose ruleOur way of choosing what to remove, against two published ones

The study ranked removals by how much every language used each entry, weighted so each language counts the same, and refused any removal that cost a language more than 0.1%. It predicted this would beat two published rules given the same additions: Purason and colleagues’ (remove the least frequent entries no merge depends on, in any script) and Didenko’s (remove the newest entries outside the protected scripts, checking that every kept entry can still be built). It did not. Purason’s frequency rule cut at least as much for all three: 17.91% against our 17.90% for Llama 3, 12.30% against 12.27% for gpt-oss, and for Qwen3 the two agree to the fourth decimal. It also made fewer texts dearer for Llama 3 (40 against 66) and gpt-oss (27 against 87); only for Qwen3 did ours make fewer (6 against 23). Didenko’s rule cut least and made the most texts dearer for all three, breaking the guard each time. A plain frequency count, with no guard at all, cut as much as our weighted, guarded rule, though for Llama 3 it also pushed Western Persian 1.45% past where it started, a second breach of the guard that ours did not make.

7 · the predictionsFive, written down first

Before any repaired vocabulary was counted on test text, the study wrote down five predictions for each of the three vocabularies, fifteen in all, with the rule that decides each. Eight held and seven failed.

8 · take one homeThe tool and the vocabularies

The procedure is a small open program, slot-ledger, MIT-licensed: download it (24 KB of Python). Give it a Hugging Face tokenizer.json and text in the languages you care about, and it writes a repaired tokenizer.json of the same size, every kept entry at its old id, with a ledger of each slot it filled, what it removed and what each language paid. A second command checks the result on text it did not see: no id moved, every new entry reachable, the reference library agreeing with it on every text, every text decoding back to itself as often as with the original, and every piece of ASCII, Latin, punctuation or emoji encoding exactly as before.

The six repaired vocabularies measured here, each a gzipped tokenizer.json with its ledger (every entry added, with its id and the merge that makes it, and every entry removed). The Llama 3 ones are for Llama 3 itself: both fill five reserved ids that Meta’s code for Llama 3.1 and later gives jobs, among them <|eom_id|> and <|python_tag|> for tool calls.

• Llama 3, 100 slots, nothing removed, no text dearer: tokenizer, ledger
• Llama 3, 749 slots, broke the guard for Kashmiri: tokenizer, ledger
• gpt-oss, 1,000 slots, nothing removed, no text dearer: tokenizer, ledger
• gpt-oss, 2,470 slots, broke the guard for Yiddish: tokenizer, ledger
• Qwen3, 1,000 slots, every guard held: tokenizer, ledger
• Qwen3, 5,000 slots, every guard held: tokenizer, ledger

Two warnings travel with them. A language you leave out is not guarded: in a test with four languages, Chinese, which was not among them, rose 0.4%. And a repaired vocabulary is a tokenizer, not a model. A model trained with the original reads every reused id with its old meaning until its rows are retrained, and the new entries’ rows start empty. Keeping the size keeps the model’s shape. It does not spare the training.

9 · better, or only cheaper?The question this page does not answer yet

Fewer tokens is cheaper. Whether it is also better is a separate question, and two studies this year point opposite ways. Chaudhary (arXiv:2610.33016) extended Llama 3’s and Qwen3’s vocabularies for Nepali and kept training small models: the original vocabulary, with no new entries, gave the best held-out Nepali. There the old rows arrive trained and the new ones do not. Regmi and colleagues (arXiv:2608.26449) trained three 268-million-parameter models that differed only in how they split words: the better-fitting split gave 4.43% fewer bits per byte on held-out Nepali at equal compute. Part two of this study trains small models from scratch on the same text with Llama 3’s vocabulary as shipped and repaired, in 24 languages, and asks whether the repaired one predicts unseen text better per byte, and what the languages it did not touch pay. Its four predictions were written down at 11:02 UTC on 30 October, after a pilot measured how much one training run differs from the next (about 0.012 bits per byte for one language, 0.006 for the average of sixteen), and before any model saw a repaired vocabulary: the pre-registration and its data file are served here. The models are training; the result will be added to this page, whichever way it goes.

10 · the edgesWhat this does not show

• A count is not a price, and fewer tokens is not understanding. The measure is tokens for the same meaning. Section 9 is about the rest.
• FLORES is one kind of text. Its 1,012 sentences per language are translated, encyclopedic and newsy. The web slices are rougher, and section 5 shows the two can disagree.
• The pre-tokeniser sets a floor no merge can pass. Llama 3 and Qwen3 cut text into words before any merge applies, with a rule that counts only letters as word characters. In Indian and other Brahmi-derived scripts, vowel signs are marks, not letters, so words are split at every vowel sign before the vocabulary sees them (Regmi et al.; Chaudhary). A repair inside the vocabulary cannot join what that rule has already cut, which is part of why Hindi, Tamil and their neighbours gain less here than scripts that are simply missing.
• The text that guarded a language is only as good as the web’s text in it. Every language’s guard came from web text (FineWeb-2, and FineWeb for English), and for some it is thin or not that language at all; section 5 is the example.
• Three vocabularies, three repairs. Other vocabularies hold different spare ids and different deadwood. The tool runs on any byte-level vocabulary saved as tokenizer.json; the numbers here are for these three.

curl -sO https://artwaste.land/checks/verify-what-a-slot-buys.mjs
node verify-what-a-slot-buys.mjs