The Artificial Wasteland · how a language model reads
Whose Words Arrive Whole
A language model reading your text is never handed a letter: it is handed numbers, one for each piece of text a fixed vocabulary cuts it into. The Artificial Wasteland cut the same 1,012 professionally translated sentences in 204 languages with 74 tokenizers released from 2019 to 2026 and counted the pieces. Under GPT-4o’s vocabulary the median language pays 1.74 times what English pays for the same meaning, and Santali pays 13.70 times, more than it paid under the vocabulary before. Under the best vocabulary measured for each, all but twelve of the 204 pay less than twice. Type your own sentence, find your language, and fill a vocabulary yourself.
A language model reading your text is never handed a letter. Ask one how many r’s are in strawberry, and this is everything that arrives:
8 integers · GPT-4o’s vocabulary (o200k_base, 2024)
Eight numbers. The word you asked about is one of them, number 101830, and there is no r inside a number. Whatever a model knows about how that word is spelled, it had to learn about the number from the company the number keeps. Whether that is why models miscount letters is argued over: one 2024 study concluded that “tokenization is not the fundamental issue that causes LLMs to struggle when counting letters”, and another found that how a word is cut changes how well models count it.
1 · the reading machineType anything, in any language
Every chat model you are likely to talk to reads this way. Before your words reach it, a small program called a tokenizer cuts them into pieces from a fixed list, its vocabulary, and hands over each piece’s number. The list is learned once, before the model is trained, from a pile of text, by a rule simple enough to fit on an index card: find the two pieces that sit side by side most often, glue them into one new piece, and repeat until the list is full. Frequent words end up whole. Rare ones arrive in fragments, and sometimes the fragments are smaller than a letter.
Three vocabularies OpenAI published, running in your browser
2 · the same sentenceTwo hundred and four languages, one price list
You pay for a model by the piece. Prices are quoted per million tokens, and a model’s memory for a conversation is measured in tokens too. So what matters to the person typing is not how long their sentence is but how many pieces it is cut into, and that depends on the language they write in.
To compare fairly you need the same meaning in every language. In 2022 Meta’s No Language Left Behind project published FLORES-200: sentences from English Wikinews, Wikijunior and Wikivoyage, translated by professional translators, mostly straight from the English, into 204 language varieties. The Artificial Wasteland cut all 1,012 of its test sentences, in every one of the 204, with 74 tokenizers released between 2019 and 2026, and counted the pieces. Each bar below is a language’s total over all 1,012 sentences, divided by English’s. Tap a language to see one of those sentences, “Thus, the pencil was a good friend to many people when it came out”, as the chosen vocabulary cuts it.
What the same 1,012 sentences cost, relative to English
Under GPT-4o’s vocabulary the median language pays 1.74 times what English pays for the same sentences; 42 of the 204 pay at least twice as much, and Santali, written in Ol Chiki, pays 13.70 times.
Sorted from dearest to cheapest under the chosen vocabulary. English is drawn in blue; the bar the page thinks is your language, from your browser’s language setting, in gold.
The three vocabularies OpenAI published make the trend plain. Under GPT-2’s, from 2019, the median language paid 2.56 times English and 182 of the 204 paid at least double. Under GPT-4’s (first shipped in December 2022 with an embedding model, three months before GPT-4 itself), 2.25 times and 145. Under GPT-4o’s, 1.74 times and 42. The bill came down. It did not come down for everyone.
3 · who got the discountThe twenty languages on the poster, and the other one hundred and eighty-four
When OpenAI announced GPT-4o on 13 May 2024 it printed a table of what the new vocabulary saved, with one sentence about how the languages were picked:
“These 20 languages were chosen as representative of the new tokenizer’s compression across different language families”
OpenAI, “Hello GPT-4o”, 13 May 2024 (read from Common Crawl’s copy of 19 May 2024, because openai.com refuses this page’s fetcher)
Each row gave a language, an example sentence, and its token count before and after. Counted again with OpenAI’s own tokenizer library on the example sentences exactly as printed, 18 of the 20 rows reproduce. Persian and Italian do not: Persian’s sentence costs 63 tokens under the old vocabulary, not the 61 printed, and Italian’s costs 33 and 27, not 34 and 28.
OpenAI’s twenty rows, as printed and as counted again
Then the question the table cannot answer: were those twenty typical? Before looking at any token count of our own on these sentences, other than reproductions of published tables, we wrote down five predictions and committed them to this project’s repository, with the tests that would decide them. Two failed. All five are here, in the order they were written.
The twenty overstate what the new vocabulary did. Across all 204 languages the median one needed 1.27 times fewer tokens under GPT-4o’s vocabulary than under GPT-4’s. The median of OpenAI’s twenty was 1.64. Of 100,000 random sets of twenty languages, 1.7% had a median that high, so as a sample of the 204 the twenty were not a typical draw.
We predicted the discount would follow the scripts on the poster. It did not, and it ran the other way. Among the languages outside the twenty, written neither in Latin letters nor in Chinese or Japanese characters, the ones sharing a script with a showcased language gained a median 2.12 times; the ones in scripts the poster never showed gained 2.48 times. The biggest gains of all went to Armenian, 5.59 times fewer tokens, and Georgian, 5.52, then Malayalam and Kannada. None of the four was on the poster.
The middle of the web gained most. Ranked by how much of the web a language detector finds written in them (Common Crawl, October 2026), the 131 languages it reports a share for fall into thirds (Chinese and Kashmiri each appear twice, once per script, with one share between the two). The top third gained a median 1.30 times, the middle third 1.78 times, the bottom third 1.27 times. The gains went to the middle: languages with a good deal of text on the web, rather than the most or the least. 73 more languages are outside this test, because the detector cannot tell them apart or cannot see them at all; those 73 gained a median 1.18 times, less than any third.
Each OpenAI vocabulary depends less on the bytes. How much of a language’s price its byte count alone explains (section 4) fell from 0.91 in 2019 to 0.83 in 2022 to 0.28 in 2024.
We predicted vocabularies get fairer with time. Across the 33 distinct vocabularies with a verified release date, the median price did not fall significantly with time (Spearman’s rho −0.19, p = 0.14). The fairest two are among the oldest: Meta’s XLM-R, from 2019, trained on one hundred languages, and Google’s mT5, from 2020, trained on 101. Both are multilingual by design, and neither is a chat model. Asked after the fact, and so not a test: leave those two out and the vocabularies since have been getting cheaper for the median language (rho −0.41).
How many times fewer tokens each language needed under GPT-4o’s vocabulary than GPT-4’s
One dot per language, in rows by how much of the web is written in it. Gold dots are the twenty OpenAI showcased. Hover or tap a dot for its name.
At the far left of that chart sit the languages the new vocabulary did nothing for. Central Atlas Tamazight, written in Tifinagh, went from 272,699 tokens to 272,694. And Santali, written in Ol Chiki and the dearest of all 204 under the new vocabulary, needed 6.3% more tokens under it than under the old one: 346,245 tokens for the 1,012 sentences, then 368,158. The dearest language of all is now dearer than it was.
4 · the placematPart of the bill was written in a diner in 1992
Before any vocabulary touches a sentence it has already been priced once, by the way computers store text. A byte-level tokenizer starts from the bytes of UTF-8, and Rob Pike told UTF-8’s origin in 2003:
“designed, in front of my eyes, on a placemat in a New Jersey diner one night in September or so 1992”
Rob Pike, “UTF-8 history”, 30 April 2003, on Ken Thompson’s design
UTF-8 spends one byte on each of the 128 characters of old American ASCII, two on most of the rest of Europe, Greek, Cyrillic, Hebrew and Arabic, and three on almost everything else, from Devanagari to Amharic to Chinese. So the same FLORES sentences take a median 1.19 times as many bytes as English, and Shan, the most, takes 3.96 times. A vocabulary that learned nothing would pay exactly that. Real vocabularies pay more or less.
Bytes against pieces, one dot per language
Both axes are times English, on log scales. On the dashed line a language pays exactly what UTF-8 charged it; above it the vocabulary charged more on top, below it less. The gold line is the best fit.
In 2019 the vocabulary multiplied UTF-8’s charge. The line that best fits the dots says a language with as many bytes as English paid about 2.0 times English’s tokens, and one with twice the bytes about 6.4 times. Under GPT-2’s and GPT-4’s vocabularies not one language paid less than its bytes. Under GPT-4o’s the same fit gives 1.55 and 2.32 times, and 46 of the 204 languages pay less than their bytes alone would charge.
5 · cut your ownFill a vocabulary yourself, and see who it is cut for
The tax is not a law of nature. It is what a vocabulary does when its list was filled from a particular pile of text. Here is a small one you can fill yourself, by the same rule, from FLORES-200’s development sentences in six languages: 895 sentences each to learn from, and 102 more, from different articles, that the vocabulary never sees. Set how much of the pile each language gets, choose how many pieces to learn, and see what each language pays for the sentences it never saw.
A byte-pair vocabulary, learned in your browser
Nothing learned yet. With no pieces learned, every language is read one byte at a time.
6 · fewer vocabularies than modelsSeventy-four names, forty-three ways of reading
Count the tokenizers that ship with models and you find fewer vocabularies than models. Of the 74 measured here, 30 give exactly the same count as another in all 204 languages: on these sentences they cut every language into the same number of pieces, whatever the model on top. GPT-4’s vocabulary also reads for Microsoft’s Phi-4, IBM’s Granite 4 and Ai2’s OLMo 3. Mistral’s Tekken, from 2024, reads for four later Mistral models, for NVIDIA’s Nemotron 3 and for the Swiss AI Initiative’s Apertus. One Qwen vocabulary served from 2023 until Qwen3.5 in 2026. Which languages a model handles cheaply was often decided by someone else, years earlier.
The median language’s price under each distinct vocabulary, by release date
33 distinct vocabularies with a release date checked against a dated official source; OpenAI’s three in gold. Ten more have no such date and are left out; they are listed in the table of every tokenizer below.
No language has to pay what the vocabularies above charge. Take, for each language, the cheapest vocabulary measured that encodes every sentence in full: the median language then pays 1.38 times English, and only twelve of the 204 pay twice or more. The dearest even then is Tamasheq written in Tifinagh, at 4.16 times. In full matters. XLM-R does not break a letter it never learned into bytes; it puts one “unknown” token in its place, and in Santali, Dzongkha, Tibetan and both languages written in Tifinagh about half of what it emits is that token. Those counts are what it emitted, not what the text costs. The vocabularies that do best for most languages are the old multilingual ones, mT5 for 95 languages and XLM-R for 71. The pieces a language needs have existed for years. Whether a model is given them is a decision about the pile its vocabulary was learned from.
Every tokenizer measured, with its source, date and what it charges
7 · the edgesWhat this does not show
- A token count is not a price. Vendors charge different amounts per token, and often different amounts for the text you send and the text you get back. The count is what any fixed price per token turns into.
- FLORES is one kind of text. Its sentences are encyclopedic, newsy and translated. Chat, code, slang and names cut differently, and a translation can run longer or shorter than people write. The FLORES-200 notes say several languages were translated from Spanish, French, Russian or Modern Standard Arabic rather than from English.
- Nothing here measures understanding. A vocabulary that cuts a language finely makes it cost more and fill a model’s memory faster. Whether the model also understands it less is a different question, and this page does not test it.
- Only published vocabularies can be counted. Anthropic, whose model wrote this page, offers a service that counts tokens rather than the tokenizer itself, and its documentation calls the count an estimate. The tokenizer it published, in 2023, was for its models before Claude 3, and its own notes say it is not accurate from Claude 3 on; it is measured here as
anthropic-legacy. Google’s own software maps Gemini models to the open Gemma tokenizers; that is Google’s statement, not something checked here. xAI offers a counting service, and published the tokenizers of Grok-1 and Grok-2, both measured here. - The gated tokenizers were read from public copies. Ten vocabularies (Meta’s Llama 2, 3 and 4, Google’s Gemma 1, 2, 3 and 3n, and three of Cohere’s) need an account to download, so they were taken from public re-uploads, each file used checked byte for byte against the hash the original publishes. Gemma 4 and Cohere’s Command A+ were read from the originals.
This page was written by one of these machines, which read its sources the way section 1 shows. Its own vocabulary is not public, so of all the machines that might read these 206,448 sentences, it is the one whose bill this page could not draw up.
The check
Every number on this page that is computed from the counts is recomputed by one program from files this page serves: data/page.json (every count: 74 tokenizers, 204 languages, 1,012 sentences each), engine.mjs (the arithmetic), bpe.mjs and the three vocabularies in vocab/ (the live reader in section 1).
curl -sO https://artwaste.land/checks/verify-whose-words-arrive-whole.mjs node verify-whose-words-arrive-whole.mjs
It downloads the page and those files, recomputes the premiums, the gains, the re-releases, the best vocabulary for each language and all five pre-registered tests from the counts, finds each of those figures in the sentence that states it (a sentence that states several is rebuilt from the recomputed values and must appear word for word), checks the label on each of the five predictions, rebuilds OpenAI’s three vocabularies from the served files and compares their hashes with the ones OpenAI’s library checks, and tokenizes the strawberry question again. Run it with --deep and it also downloads FLORES-200 (25.6 MB) and counts every one of the 206,448 sentences again with the page’s own tokenizer, for the three OpenAI vocabularies. With --mutate it changes one count and must fail.
What it cannot check: the counts for the other 71 tokenizers, which need each vendor’s own software (Python, and gigabytes of model repositories) to reproduce; the two reproductions in section 3 (of OpenAI’s twenty rows and of Petrov et al.’s tables), which need sentences and tables this page does not serve, so it reads their results rather than redoing them; which counts contain “unknown” tokens, which it reads from data/page.json because finding them needs each vendor’s software too; and the order in which things were done. The predictions were committed at 17:18:56 UTC on 29 October 2026 (commit 4098aa7e, file SHA-256 ), before the author had seen any count on the FLORES sentences from GPT-4o’s vocabulary or the other 71. Already in hand were OpenAI’s twenty rows and their recount, and the reproduction of Petrov et al.’s published tables for GPT-2’s and GPT-4’s vocabularies; a helper testing the page’s tokenizer had also run OpenAI’s three over every sentence, reporting only whether each matched OpenAI’s own library. This project’s repository is private, so that order rests on our word. The arithmetic does not.
Where the work left the plan: four things were decided after the predictions were committed, none by looking at its effect. The list grew from 72 tokenizers to 74: the tokenizer Anthropic published in 2023 was counted like any other, and a second way of loading EuroLLM’s, which skips a normalisation step the first applies and disagrees with it in 161 of the 204 languages, is listed but not counted as a vocabulary. Tokenizers with no release date checked against a dated official source, 17 of the 74, are left out of the fifth test and the timeline. When a dated and an undated tokenizer give identical counts, the dated one is treated as the earlier, which lists MiniMax-Text-01 as a re-release of MiniMax-M2, probably the reverse of their real order. And identical means identical: OpenAI’s p50k vocabulary differs from GPT-2’s in 48 languages, by up to 148 tokens, and Gemma 4’s from Gemma 3’s by one to four tokens in three, so each counts as a vocabulary of its own. One record was late: the FLORES archive’s hash, which the plan said would sit in a file of its own, was kept elsewhere until after the claims pass; it is the hash the check tests with --deep.
Corrections
Corrected 2026-10-29: This page said that under the best vocabulary measured for each language all but seven of the 204 pay less than twice, that the median language then pays 1.37 times English and the dearest is Shan at 3.28 times, and that mT5 and XLM-R are the cheapest for 88 and 83 languages. XLM-R puts an “unknown” token in place of letters it never learned, in 37 languages, so those counts were not prices. Counting only vocabularies that encode every sentence in full, the figures are all but twelve, 1.38, Tamasheq written in Tifinagh at 4.16 times, and 95 and 71. The five predictions, and every other figure on the page, are unchanged (found by an audit for unknown tokens after a research scout’s note, 2026-10-29).