---
slug: whose-words-arrive-whole
passed: 2026-10-29
by: ccr-fc751e34-gmn18l, one agent on oversight/claims-pass.md (first pass at 601f1357, re-check of the fixes at 39849f03; layer not yet deployed, reader check on local copies); every finding re-read against its source by the author before fixing
checked: 102
confirmed: 70
wrong: 3
unverifiable: 0
observed: 29
stale: 0
---
# Claims pass: Whose Words Arrive Whole

Live page not checked: the layer was not deployed at the time (404). Reader check on local copies (the eight
files the verifier names, in an empty directory; the verifier in a second one). First pass, at 601f1357:
`AW_DIR=<dir> node verify-whose-words-arrive-whole.mjs` exit 0, 99/0; `--mutate` exit 1; `--deep` 105/0.
Re-check at 39849f03: 147/0, `--mutate` exit 1, `--deep` 153/0. `node assay/sources.mjs
whose-words-arrive-whole --online` after the fixes: every excerpt found in its kept copy; 11 NOT FOUND online,
all arXiv abstract pages (the excerpt is in the PDF, read there) or OpenAI's RSS feed (the tool strips CDATA;
the dates were read in the live feed). The page prints no verifier transcript; the check was tested by single
edits to the page's sentences (the agent's tools/mutate.mjs) and, after the second round of fixes, by a sweep
that changed the last digit of every number in index.html's text and meta copies, one at a time (the first
such sweep, after the re-check's fixes, caught 225 of 369; its survivors included data-derived years, dates
and counts the check did not yet read, which it now reads; the second caught 272 of 369, and each of the 97
survivors is page metadata (viewport, colour, image size, JSON-LD dates), a section number, a digit inside a
model or dataset name (GPT-2, Claude 3, FLORES-200), the trainer's default of 3,000, a number inside a quoted
source or citation (checked against its source by this pass), or the archive's 25.6 MB, which --deep reads).
After all fixes the verifier reads 217 passed, 0 failed; `--mutate` fails 20 checks; `--deep` from an empty
directory 224 passed, 0 failed.

## Claims
- CONFIRMED | In 2022 Meta's No Language Left Behind project published FLORES-200 | https://arxiv.org/pdf/2207.04672 | NLLB Team (Meta AI), arXiv 2207.04672, 2022; model card "Developed by Meta AI Research"; flores README changelog "2022-06-30: released flores-200" (excerpt is in the PDF, not the abs page, so the tool reads NOT FOUND)
- CONFIRMED | sentences from English Wikinews, Wikijunior and Wikivoyage | https://arxiv.org/pdf/2207.04672 | "3001 sentences sampled from English-language Wikimedia projects ... Approximately one third of sentences are collected from each of these sources: Wikinews, Wikijunior, and Wikivoyage"
- CONFIRMED | translated by professional translators | https://arxiv.org/pdf/2207.04672 | "Flores-200 is created with professional human translators who translate the FLORES source dataset into the target languages"
- CONFIRMED | into 204 language varieties | https://arxiv.org/pdf/2207.04672 | "for 204 total languages"; split table dev 997, devtest 1012
- CONFIRMED | The FLORES-200 notes say several languages were translated from Spanish, French, Russian or Modern Standard Arabic rather than from English. | https://github.com/facebookresearch/flores/blob/main/flores200/README.md | "Specifically, several languages were translated from Spanish, French, Russian and Modern Standard Arabic."
- CONFIRMED | mostly straight from the English | https://github.com/facebookresearch/flores/blob/main/flores200/README.md | "FLORES-200 has several languages which were not translated from English." Several, so most were from English
- CONFIRMED | the sentences shown here and the development sentences in section 5 are FLORES-200 text, licensed CC BY-SA 4.0 | https://github.com/facebookresearch/flores | "FLORES-200: CC-BY-SA 4.0"
- CONFIRMED | Prices are quoted per million tokens | https://docs.x.ai/developers/pricing.md | "Prices shown per million tokens." (record URL corrected in 39849f03; the online tool now reads FOUND)
- CONFIRMED | Prices are quoted per million tokens | https://platform.claude.com/docs/en/about-claude/pricing.md | "MTok: Million tokens. $5 / MTok is $5 for every million tokens." (record URL corrected; FOUND)
- CONFIRMED | a model's memory for a conversation is measured in tokens too | https://platform.claude.com/docs/en/build-with-claude/context-windows | the context window "holds the conversation history plus the new output Claude generates", sized in tokens
- CONFIRMED | find the two pieces that sit side by side most often, glue them into one new piece, and repeat until the list is full | https://www.derczynski.com/papers/archive/BPE_Gage.pdf | Gage: "finding the most frequently occurring pairs of adjacent bytes in the data and replacing all instances of the pair ... repeats this process"; Sennrich 2016: final vocabulary size is the initial one plus the number of merges
- CONFIRMED | Philip Gage, "A New Algorithm for Data Compression", The C Users Journal, February 1994, the origin of byte-pair encoding | https://arxiv.org/pdf/1508.07909 | reference list: "Philip Gage. 1994. A New Algorithm for Data Compression. C Users J., 12(2):23-38, February."
- CONFIRMED | Sennrich, Haddow and Birch, ACL 2016, which adapted it to splitting words | https://aclanthology.org/P16-1162/ | Proceedings of the 54th Annual Meeting of the ACL (2016); "We adapt byte pair encoding (BPE) (Gage, 1994), a compression algorithm, to the task of word segmentation."
- CONFIRMED | Radford et al., "Language Models are Unsupervised Multitask Learners", 2019, the byte-level version | https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf | title and authors match; "a byte-level version of BPE only requires a base vocabulary of size 256"
- CONFIRMED | one 2024 study concluded that "tokenization is not the fundamental issue that causes LLMs to struggle when counting letters" | https://arxiv.org/abs/2412.18626 | verbatim in Fu et al., arXiv 2412.18626v1, 19 Dec 2024
- CONFIRMED | another found that how a word is cut changes how well models count it | https://arxiv.org/abs/2410.19730 | Zhang, Cao, You 2024: "uncovering substantial performance variations based on input tokenization differences" (their test strings are runs of letters, not dictionary words; the substance holds)
- CONFIRMED | When OpenAI announced GPT-4o on 13 May 2024 | https://openai.com/news/rss.xml | feed item "Hello GPT-4o", pubDate Mon, 13 May 2024 10:05:00 GMT (fetched live; the online tool strips CDATA as a tag, hence NOT FOUND)
- CONFIRMED | These 20 languages were chosen as representative of the new tokenizer's compression across different language families | https://openai.com/index/hello-gpt-4o/ | verbatim in Common Crawl CC-MAIN-2024-22 captures of 2024-05-19T13:30:46Z, 2024-05-24 and 2024-05-26, fetched independently by range request from data.commoncrawl.org (openai.com answers 403)
- CONFIRMED | Each row gave a language, an example sentence, and its token count before and after. | https://openai.com/index/hello-gpt-4o/ | 20 rows, identical in all three captures, e.g. "Gujarati 4.4x fewer tokens (from 145 to 33)" with its example sentence
- CONFIRMED | first shipped in December 2022 | https://openai.com/news/rss.xml | "New and improved embedding model", pubDate Thu, 15 Dec 2022 08:00:00 GMT (fetched live)
- CONFIRMED | with an embedding model | https://github.com/openai/tiktoken/blob/main/tiktoken/model.py | "text-embedding-ada-002": "cl100k_base" (fetched live)
- CONFIRMED | three months before GPT-4 itself | https://openai.com/news/rss.xml | two feed items titled GPT-4, both Tue, 14 Mar 2023; 15 Dec 2022 to 14 Mar 2023 is three months
- OBSERVED | Counted again with OpenAI's own tokenizer library on the example sentences exactly as printed, 18 of the 20 rows reproduce. | recreated: Python tiktoken 0.14.0 on the 20 sentences extracted from the 2024-05-19 Common Crawl capture (research/whose-words-arrive-whole/data/r2-openai20.json is the record) | 18 of 20 equal the printed counts
- OBSERVED | Persian's sentence costs 63 tokens under the old vocabulary, not the 61 printed, and Italian's costs 33 and 27, not 34 and 28. | recreated: Python tiktoken 0.14.0 on the captured sentences (record research/whose-words-arrive-whole/data/r2-openai20.json) | Persian 63 then 32 (printed 61 then 32); Italian 33 then 27 (printed 34 then 28)
- CONFIRMED | designed, in front of my eyes, on a placemat in a New Jersey diner one night in September or so 1992 | https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt | verbatim: "UTF-8 was designed, in front of my eyes, on a placemat in a New Jersey diner one night in September or so 1992."
- CONFIRMED | Rob Pike told UTF-8's origin in 2003 | https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt | "Subject: UTF-8 history ... Date: Wed, 30 Apr 2003 22:32:32 -0700"
- CONFIRMED | on Ken Thompson's design | https://www.rfc-editor.org/rfc/rfc3629 | "UTF-8 was devised in September 1992 by Ken Thompson, guided by design criteria specified by Rob Pike"; Pike: "Ken figured out the bit-packing"
- CONFIRMED | UTF-8 spends one byte on each of the 128 characters of old American ASCII, two on most of the rest of Europe, Greek, Cyrillic, Hebrew and Arabic, and three on almost everything else, from Devanagari to Amharic to Chinese. | https://www.rfc-editor.org/rfc/rfc3629 | section 3: U+0000 to 007F one octet, U+0080 to 07FF two (Greek, Cyrillic, Hebrew, Arabic), U+0800 to FFFF three (Devanagari, Ethiopic, CJK)
- CONFIRMED | F. Yergeau, RFC 3629, 2003 | https://www.rfc-editor.org/rfc/rfc3629 | "F. Yergeau ... Request for Comments: 3629 ... November 2003"
- CONFIRMED | trained on one hundred languages | https://arxiv.org/abs/1911.02116 | "We train a Transformer-based masked language model on one hundred languages"; submitted 5 Nov 2019
- CONFIRMED | trained on 101 | https://arxiv.org/abs/2010.11934 | "covering 101 languages"; submitted 22 Oct 2020
- CONFIRMED | Both are multilingual by design, and neither is a chat model. | https://arxiv.org/abs/1911.02116 | XLM-R is a masked language model; mT5 (arXiv 2010.11934) a text-to-text variant of T5
- CONFIRMED | offers a service that counts tokens rather than the tokenizer itself | https://platform.claude.com/docs/en/build-with-claude/token-counting | "Token counting lets you determine the number of tokens in a message before you send it to Claude."
- CONFIRMED | its documentation calls the count an estimate | https://platform.claude.com/docs/en/build-with-claude/token-counting | "The token count is an **estimate**."
- CONFIRMED | was for its models before Claude 3, and its own notes say it is not accurate from Claude 3 on | https://github.com/anthropics/anthropic-tokenizer-typescript | "As of the Claude 3 models, this algorithm is no longer accurate, but can be used as a very rough approximation."
- CONFIRMED | The tokenizer it published, in 2023 | https://registry.npmjs.org/@anthropic-ai/tokenizer | "0.0.1":"2023-06-28T23:07:02.849Z"
- CONFIRMED | Google's own software maps Gemini models to the open Gemma tokenizers; that is Google's statement, not something checked here. | https://raw.githubusercontent.com/googleapis/python-genai/main/google/genai/_local_tokenizer_loader.py | "gemini-2.5-pro": "gemma3", and the same for 2.5 Flash, Flash-Lite and 2.0 Flash (record URL corrected; FOUND)
- CONFIRMED | xAI offers a counting service | https://docs.x.ai/developers/rest-api-reference/inference/other.md | "POST /v1/tokenize-text Tokenize text with the specified model" (record URL corrected; FOUND)
- CONFIRMED | published the tokenizers of Grok-1 and Grok-2, both measured here | research/whose-words-arrive-whole/tok/manifest.json | pinned github xai-org/grok-1@7050ed20 and Hugging Face xai-org/grok-2@daf4395a, both counted in data/totals.json
- OBSERVED | they were taken from public re-uploads, each file used checked byte for byte against the hash the original publishes | research/whose-words-arrive-whole/tok/manifest.json | record: all 18 files of the 10 mirrored tokenizers carry matches_original true
- CONFIRMED | the library is MIT-licensed | https://github.com/openai/tiktoken/blob/main/LICENSE | "MIT License Copyright (c) 2022 OpenAI, Shantanu Jain"
- CONFIRMED | Petrov, La Malfa, Torr and Bibi, "Language Model Tokenizers Introduce Unfairness Between Languages", NeurIPS 2023, arXiv:2305.15425 | https://arxiv.org/abs/2305.15425 | authors Petrov, La Malfa, Torr, Bibi; comments "Published at NeurIPS 2023"
- OBSERVED | our counting reproduces their published totals for GPT-2 and GPT-4's vocabularies in all 204 languages | recreated: python3 -I research/whose-words-arrive-whole/analysis/reproduce_petrov.py with tiktoken 0.14.0 against Petrov's CSV fetched from GitHub (sha256 68ef66ea, equal to the kept copy) | 204 of 204 for GPT-2, r50k_base and cl100k_base; output identical to the committed data/r1-petrov.json
- CONFIRMED | Ten vocabularies (Meta's Llama 2, 3 and 4, Google's Gemma 1, 2, 3 and 3n, and three of Cohere's) need an account to download | research/whose-words-arrive-whole/tok/manifest.json | meta-llama2/3/4, google-gemma/gemma2/gemma3/gemma3n, cohere-command-r/aya-expanse/tiny-aya: originals gated (manual or auto), read from public copies (mirror_used true); ten
- CONFIRMED | Gemma 4 and Cohere's Command A+ were read from the originals. | research/whose-words-arrive-whole/tok/manifest.json | google/gemma-4-E4B-it and CohereLabs/command-a-plus-05-2026-bf16: gated false, mirror_used false
- OBSERVED | Already in hand were OpenAI's twenty rows and their recount, and the reproduction of Petrov et al.'s published tables for GPT-2's and GPT-4's vocabularies | git show 4098aa7e:research/whose-words-arrive-whole/PREREGISTRATION.md | its "Seen" list names OpenAI's 20 rows recounted (18 of 20) and Petrov's r50k and cl100k tables reproduced (204 of 204); it also lists byte and code point totals and an unread cl100k smoke test
- OBSERVED | a helper testing the page's tokenizer had also run OpenAI's three over every sentence, reporting only whether each matched OpenAI's own library | research/whose-words-arrive-whole/tokenizer-js/test-output.txt | record: match tallies only ("206448 / 206448" for each vocabulary), no counts. Timing not in git (first committed aae7679, 17:37); filesystem mtime and ctime both 2026-10-29 17:12:41.4566 UTC (ctime equal to mtime, so not re-touched), written in the same instant as four sibling files, and before this checkout's FLORES archive arrived (17:13:11), so the 184 s run happened elsewhere and its output was copied in 6 min 15 s before the 17:18:56 commit. Supported, on a local record a reader cannot see
- CONFIRMED | The list grew from 72 tokenizers to 74 | git show 4098aa7e:research/whose-words-arrive-whole/PREREGISTRATION.md | "72 at the time of writing"; data/page.json has 74 tokenizers; DEVIATIONS.md 1 names anthropic-legacy and utter-eurollm-hf as the two added
- CONFIRMED | Ranked by how much of the web a language detector finds written in them (Common Crawl, October 2026) | https://commoncrawl.github.io/cc-crawl-statistics/plots/languages.csv | CSV sha256 261dfd2d equals the provenance record; CC-MAIN-2026-43 is the "October 2026 Crawl Archive"
- OBSERVED | The predictions were committed at 17:18:56 UTC on 29 October 2026 | git show 4098aa7e | AuthorDate and CommitDate 2026-10-29T17:18:56+00:00; PREREGISTRATION.md sha256 4135a0d6 then and now, equal to data/page.json preregistration.sha256
- OBSERVED | Under GPT-4o's vocabulary the median language pays 1.74 times what English pays for the same sentences | recreated with Python tiktoken 0.14.0 over all 206,448 devtest sentences (local archive sha256 b8b0b767): 612 of 612 counts for r50k, cl100k and o200k equal data/page.json | verifier arithmetic: 1.74
- OBSERVED | Santali, written in Ol Chiki, pays 13.70 times | recreated with Python tiktoken 0.14.0 over all 206,448 devtest sentences (local archive sha256 b8b0b767): 612 of 612 counts for r50k, cl100k and o200k equal data/page.json | Santali the dearest, 13.70
- OBSERVED | Under GPT-2's, from 2019, the median language paid 2.56 times English and 182 of the 204 paid at least double. | recreated with Python tiktoken 0.14.0 over all 206,448 devtest sentences (local archive sha256 b8b0b767): 612 of 612 counts for r50k, cl100k and o200k equal data/page.json | 2.56 and 182; GPT-4 2.25 and 145; GPT-4o 1.74 and 42
- OBSERVED | Across all 204 languages the median one needed 1.27 times fewer tokens under GPT-4o's vocabulary than under GPT-4's. | recreated with Python tiktoken 0.14.0 over all 206,448 devtest sentences (local archive sha256 b8b0b767): 612 of 612 counts for r50k, cl100k and o200k equal data/page.json | median gain 1.27; showcase 1.64; p_upper 0.0166
- OBSERVED | The biggest gains of all went to Armenian, 5.59 times fewer tokens, and Georgian, 5.52, then Malayalam and Kannada. | recreated with Python tiktoken 0.14.0 over all 206,448 devtest sentences (local archive sha256 b8b0b767): 612 of 612 counts for r50k, cl100k and o200k equal data/page.json | Armenian 5.59, Georgian 5.52, then Malayalam, Kannada; none in the twenty
- OBSERVED | The top third gained a median 1.30 times, the middle third 1.78 times, the bottom third 1.27 times. | recreated with Python tiktoken 0.14.0 over all 206,448 devtest sentences (local archive sha256 b8b0b767): 612 of 612 counts for r50k, cl100k and o200k equal data/page.json; CC proxy CSV re-fetched | 1.30, 1.78, 1.27; p 0.00075 and 0.00043; also holds with 44/43/44, 44/44/43 and 43/44/44 splits
- OBSERVED | fell from 0.91 in 2019 to 0.83 in 2022 to 0.28 in 2024 | recreated with Python tiktoken 0.14.0 over all 206,448 devtest sentences (local archive sha256 b8b0b767): 612 of 612 counts for r50k, cl100k and o200k equal data/page.json; bytes recounted from the FLORES files | R squared 0.913, 0.830, 0.276; cl100k dated 2022-12-15
- OBSERVED | the median price did not fall significantly with time | research/whose-words-arrive-whole/data/totals.json (commit aae7679) | verifier recomputes rho -0.19, p 0.143 over 33 dated distinct vocabularies; of the other 71 tokenizers only Phi-4, Granite 4, Apertus and p50k were recounted here (204 of 204 each)
- OBSERVED | Central Atlas Tamazight, written in Tifinagh, went from 272,699 tokens to 272,694. | recreated with Python tiktoken 0.14.0 over all 206,448 devtest sentences (local archive sha256 b8b0b767): 612 of 612 counts for r50k, cl100k and o200k equal data/page.json | 272,699 (cl100k) and 272,694 (o200k)
- OBSERVED | 346,245 tokens for the 1,012 sentences, then 368,158 | recreated with Python tiktoken 0.14.0 over all 206,448 devtest sentences (local archive sha256 b8b0b767): 612 of 612 counts for r50k, cl100k and o200k equal data/page.json | 346,245 and 368,158, 6.3% more
- OBSERVED | So the same FLORES sentences take a median 1.19 times as many bytes as English, and Shan, the most, takes 3.96 times. | recreated: UTF-8 bytes summed from research/whose-words-arrive-whole/tokenizer-js/flores/flores200_dataset/devtest | median 1.1921, Shan 3.9553; bytes equal data/page.json in all 204
- OBSERVED | Of the 74 measured here, 30 give exactly the same count as another in all 204 languages | research/whose-words-arrive-whole/data/totals.json (commit aae7679) | verifier finds 43 distinct and 30 re-releases; "as another" now fits MiniMax-Text-01, matched to the later MiniMax-M2
- OBSERVED | GPT-4's vocabulary also reads for Microsoft's Phi-4, IBM's Granite 4 and Ai2's OLMo 3. | recreated: the pinned tokenizer.json of microsoft/phi-4@2db69c1c and ibm-granite/granite-4.0-micro@56111ae1 (sha256 as the manifest) with tokenizers 0.23.3, research/whose-words-arrive-whole/tokenizer-js/flores devtest | 204 of 204 equal data/page.json and the tiktoken cl100k recount; OLMo 3 not recounted
- OBSERVED | Mistral's Tekken, from 2024, reads for four later Mistral models, for NVIDIA's Nemotron 3 and for the Swiss AI Initiative's Apertus. | research/whose-words-arrive-whole/data/totals.json (commit aae7679) | reReleaseOf lists tekken-2501, -2506, -2512, -2603, nemotron3, apertus; Apertus recreated 204 of 204 from swiss-ai/Apertus-8B-2509@3162c996; Tekken dated 18 July 2024 (mistral.ai/news/mistral-nemo, live)
- OBSERVED | One Qwen vocabulary served from 2023 until Qwen3.5 in 2026. | research/whose-words-arrive-whole/data/totals.json (commit aae7679) | qwen2, qwen2.5, qwen3 equal qwen1 (2023-08-03); qwen3.5 distinct, dated 2026-02-16 (QwenLM/Qwen3.5 README, live)
- WRONG | the median language then pays 1.37 times English, and only seven of the 204 pay twice or more (and in the description: all but seven of the 204 pay less than twice; the dearest even then is Shan, at 3.28 times) | research/whose-words-arrive-whole/data/unk-audit.json | the arithmetic on data/totals.json is right, but the best vocabulary for 12 languages was XLM-R counting text it replaced with unknown tokens (Santali, Dzongkha, Tibetan, both Tifinagh languages about half its tokens); counting only vocabularies that encode every sentence in full: median 1.38, twelve at twice or more, dearest Tamasheq in Tifinagh at 4.16 (analysis/unk_audit.py, found after publication, 2026-10-29)
- WRONG | The vocabularies that do best for most languages are the old multilingual ones, mT5 for 88 languages and XLM-R for 83. | research/whose-words-arrive-whole/data/unk-audit.json | 12 of XLM-R's 83 rested on counts containing unknown tokens; counting only full encodings, mT5 95 and XLM-R 71 (found after publication, 2026-10-29)
- OBSERVED | The word you asked about is one of them, number 101830, and there is no r inside a number. | recreated: Python tiktoken 0.14.0 o200k_base on "How many r's are in strawberry?" | 5299, 1991, 428, 885, 553, 306, 101830, 30
- OBSERVED | disagrees with it in 161 of the 204 languages | research/whose-words-arrive-whole/data/totals.json (via public/strata/whose-words-arrive-whole/data/page.json) | utter-eurollm-hf differs from utter-eurollm in 161 of 204 languages (recomputed from page.json; the counts themselves not recreated)
- CONFIRMED | Tokenizers with no release date checked against a dated official source, 17 of the 74, are left out of the fifth test and the timeline. | public/strata/whose-words-arrive-whole/data/page.json | 17 of 74 tokenizers carry no verified date; P5 runs on the 33 dated distinct vocabularies; app.mjs draws the timeline only from dated ones
- OBSERVED | OpenAI's p50k vocabulary differs from GPT-2's in 48 languages, by up to 148 tokens, and Gemma 4's from Gemma 3's by one to four tokens in three | recreated: Python tiktoken 0.14.0 p50k_base over all devtest sentences (research/whose-words-arrive-whole/tokenizer-js/flores) | p50k 204 of 204 equal data/page.json; differs from r50k in 48 languages, by up to 148; Gemma 4 against Gemma 3 recomputed from page.json: 3 languages (hye +2, zsm +4, zul +1)
- CONFIRMED | Ahia et al., "Do All Languages Cost the Same?", EMNLP 2023 | https://aclanthology.org/2023.emnlp-main.614/ | Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing; first author Orevaoghene Ahia
- CONFIRMED | Arnett, Chang and Bergen, "A Bit of a Problem", 2024, arXiv:2403.00686, on the byte premium | https://arxiv.org/abs/2403.00686 | submitted 1 Mar 2024; "we define the byte premium between two languages as the ratio of bytes used to encode content-matched text"
- CONFIRMED | Web shares: Common Crawl, CC-MAIN-2026-43, language statistics (CLD2) | https://commoncrawl.github.io/cc-crawl-statistics/plots/languages | "the percentage covered by the primary language of a document (returned first by CLD2)"; CC-MAIN-2026-43 rows present
- CONFIRMED | The Artificial Wasteland cut all 1,012 of its test sentences, in every one of the 204 (206,448 in all) | https://arxiv.org/pdf/2207.04672 | devtest 1012 sentences; 204 x 1,012 = 206,448; the archive (sha256 b8b0b767) has 204 devtest files of 1,012 lines
- OBSERVED | 895 sentences each to learn from, and 102 more, from different articles, that the vocabulary never sees | recreated: the split rule of research/whose-words-arrive-whole/trainer/prepare_corpus.py over metadata_dev.tsv | 281 articles; held-out = article number mod 10 = 9 gives 895 and 102 sentences, no article shared; line numbers equal the shipped data/flores200-dev-6.json
- CONFIRMED | Tap a language to see one of those sentences, "Thus, the pencil was a good friend to many people when it came out" | research/whose-words-arrive-whole/tokenizer-js/flores/flores200_dataset/devtest/eng_Latn.devtest | line 644 (page.json sentenceLine 644) is that sentence
- CONFIRMED | read from Common Crawl's copy of 19 May 2024, because openai.com refuses this page's fetcher | https://data.commoncrawl.org/ (CC-MAIN-2024-22 index) | capture 2024-05-19T13:30:46Z of https://openai.com/index/hello-gpt-4o/?ref=futuretools.io exists; openai.com answered this pass with HTTP 403
- CONFIRMED | This project's repository is private, so that order rests on our word | https://github.com/artwasteland/Artificial-Wasteland | gh api: "private": true
- MINOR | Before looking at any count of our own on these sentences, other than reproductions of published tables, we wrote down five predictions | git show 4098aa7e:research/whose-words-arrive-whole/PREREGISTRATION.md | true of token counts (the smoke test was unread, the helper reported matches only), but "Seen" also lists "each language's UTF-8 byte and code point totals", which are counts of our own on these sentences; counted as confirmed
- CONFIRMED | before the author had seen any count on the FLORES sentences from GPT-4o's vocabulary or the other 71 | git show 4098aa7e:research/whose-words-arrive-whole/PREREGISTRATION.md | "Not seen: any o200k_base count on FLORES; any count from the other 69 tokenizers on FLORES" (the two added later were not yet in the inventory)
- CONFIRMED | Every number on this page that is computed from the counts is recomputed by one program from files this page serves; it reads the two reproductions' results rather than redoing them | public/checks/verify-whose-words-arrive-whole.mjs | the verifier recomputes premiums, gains, re-releases, best vocabulary and P1 to P5 from data/page.json; R1 and R2 are read from results (lines in section C), as the panel now says
- WRONG | checks that each of those figures the prose states is in its own sentence | public/checks/verify-whose-words-arrive-whole.mjs | says() tests that the true figure occurs as a substring of the sentence: an added leading digit (42 to 142 of the 204, 1.7% to 11.7%, 1.27 to 11.27, 13.70 to 113.70 in the readout, 48 to 148, 161 to 1161, 17 to 117, 2.56 to 12.56; aw-plain 1.74 to 11.74; og and twitter 74 to 174 tokenizers) and swaps within a sentence (P3 top and middle, Armenian and Georgian, Tamazight and Santali before and after, Persian and Italian, 1.55 and 2.32) leave 147 passed; restatements not read: "those 73", "the other 71" twice, the count line's "8 integers", 204 and 74 and "2019 and 2026" in section 2, "(every count: 74 tokenizers, 204 languages, 1,012 sentences each)"
- CONFIRMED | every other tokenizer was run from a pinned revision, named in the table above | public/strata/whose-words-arrive-whole/data/page.json | all 74 tokenizers carry a pin equal to tok/manifest.json (repo and revision, or file); the ten mirrored ones also name the original repo and revision; app.mjs:318 prints them
- CONFIRMED | the 131 languages it reports a share for fall into thirds (Chinese and Kashmiri each appear twice, once per script, with one share between the two) | public/strata/whose-words-arrive-whole/data/page.json | zho_Hans and zho_Hant share one ccPct, kas_Arab and kas_Deva another; 131 languages carry one
- OBSERVED | Anthropic, whose model wrote this page | git log 4098aa7e..39849f03 | record: every commit for the layer carries "Co-Authored-By: Claude Opus 5.5"
- CONFIRMED | Its own vocabulary is not public | https://platform.claude.com/docs/en/build-with-claude/token-counting | Anthropic offers counting as a service; its only published tokenizer "is no longer accurate" as of Claude 3
- CONFIRMED | Vendors charge different amounts per token, and often different amounts for the text you send and the text you get back | https://platform.claude.com/docs/en/about-claude/pricing.md | separate "Base Input Tokens" and "Output Tokens" columns
- CONFIRMED | the vocabulary files carry no licence statement of their own | research/whose-words-arrive-whole/tokenizer-js/ranks (copies of https://openaipublic.blob.core.windows.net/encodings/) | all three .tiktoken files consist only of "base64 rank" lines
- CONFIRMED | Run it with --deep and it also downloads FLORES-200 (25.6 MB) and counts every one of the 206,448 sentences again | https://dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz | content-length 25,585,843; sha256 b8b0b767 as the verifier and data/corpus.json record; --deep: 153 passed, 0 failed
- CONFIRMED | compares their hashes with the ones OpenAI's library checks | https://github.com/openai/tiktoken/blob/main/tiktoken_ext/openai_public.py | expected_hash 306cd27f, 223921b7, 446a9538, equal to bpe.mjs ENCODINGS
- CONFIRMED | P1 card: the twenty overstate what the new vocabulary did; held; not a typical draw | git show 4098aa7e:research/whose-words-arrive-whole/PREREGISTRATION.md | stated faithfully; supported if p_upper < 0.05, "not a typical draw" if below 0.025; p_upper 0.0166; label now also checked by the verifier
- CONFIRMED | P2 card: predicted the discount would follow the scripts on the poster; failed | git show 4098aa7e:research/whose-words-arrive-whole/PREREGISTRATION.md | scope and the seven shared scripts as registered; p 0.993
- CONFIRMED | P3 card: the middle of the web gained most; held | git show 4098aa7e:research/whose-words-arrive-whole/PREREGISTRATION.md | 131 in, 73 out as registered; both one-sided p < 0.05 (0.00075, 0.00043); the unregistered tercile-size convention does not change it
- CONFIRMED | P4 card: each OpenAI vocabulary depends less on the bytes; held | git show 4098aa7e:research/whose-words-arrive-whole/PREREGISTRATION.md | both inequalities required; 0.913 > 0.830 > 0.276
- CONFIRMED | P5 card: predicted vocabularies get fairer with time; failed; the after-the-fact rho -0.41 labelled as not a test | git show 4098aa7e:research/whose-words-arrive-whole/PREREGISTRATION.md | restricted to 33 dated distinct vocabularies (DEVIATIONS.md 2, stated on the card and in the new paragraph); rho -0.19, p 0.143
- CONFIRMED | release dates in the table and timeline, from each tokenizer's official announcement, model card or paper | research/whose-words-arrive-whole/tok/manifest.json | every quoted date in the 57 dated entries is in its kept snapshot; live: OpenAI RSS (14 Feb 2019, 15 Dec 2022, 13 May 2024), Mistral NeMo "July 18, 2024", Gemma 4 "Apr 02, 2026", Qwen3.5 "2026-02-16", Mistral Large 4 2026-10-29; DeepSeek-V4 2026/04/24 not visible in the live HTML (kept snapshot has it)
- OBSERVED | narration: Fill a list yourself with equal text from six languages, and most of the difference goes away | recreated: public/strata/whose-words-arrive-whole/trainer/train-bpe.mjs in Node, 3,000 merges, data/flores200-dev-6.json | equal bytes: held-out premiums 0.92 to 1.26 times English; mostly English 2.09 to 3.00; no merges 0.93 to 2.53
- CONFIRMED | a second way of loading EuroLLM's, which skips a normalisation step the first applies | research/whose-words-arrive-whole/tok/manifest.json | utter-eurollm-hf: "its tokenizer.json has no NFKC normalizer while tokenizer.model's SentencePiece normalizer is 'nfkc'"
- CONFIRMED | When a dated and an undated tokenizer give identical counts, the dated one is treated as the earlier, which lists MiniMax-Text-01 as a re-release of MiniMax-M2, probably the reverse of their real order | research/whose-words-arrive-whole/tok/manifest.json | minimax-text01 dated only by paper submission 2025-01-14 (partial), minimax-m2 verified 2025-10-27; data reReleaseOf minimax-text01 = minimax-m2 (verifier checks it)
- CONFIRMED | One record was late: the FLORES archive's hash, which the plan said would sit in a file of its own, was kept elsewhere until after the claims pass; it is the hash the check tests with --deep | git show 4098aa7e:research/whose-words-arrive-whole/PREREGISTRATION.md | plan: "sha256 recorded in data/corpus.json"; corpus.json first committed in 39849f03; the hash b8b0b767 first committed in b11ddc58 (sources/flores200-dataset-README.txt and the verifier), which --deep tests
- OBSERVED | four things were decided after the predictions were committed, none by looking at its effect | research/whose-words-arrive-whole/DEVIATIONS.md | record: DEVIATIONS.md (first committed aae7679) lists them and says "none was chosen by looking at its effect"; the four on the page match items 1 to 4; intent is not otherwise checkable

## Findings
- [fixed] HIGH-1, the description's headline figures were checked only by a whole-page includes, so the description's "13.70", "74 tokenizers" and "1,012" could change with the check green (an edit to any one of them, or to the readout's "1.74", left 99 passed). Since 39849f03 the verifier reads the description paragraph and the price-list readout on their own and takes both out of the body text, an anchor must occur exactly once, and the meta and JSON-LD descriptions must equal the description word for word; og, twitter and the plain summary are checked too. The re-check found every single-copy edit caught; the substring weakness it found in the remaining copies is NEW-1 below, also fixed.
- [fixed] MED-1, the check panel claimed more than the verifier did ("computed rather than quoted is recomputed", "each figure the prose states"): R1 and R2 are read from the shipped results, and a dozen prose figures were never read. The panel now says "computed from the counts", lists the two reproductions under what the check cannot redo (it reads their results), and every figure the first pass listed is now checked; the re-check confirmed each one fails when changed.
- [fixed] MED-2, "Before counting anything" and "before the author had seen any count from GPT-4o's vocabulary or the other 71" overstated what was unseen: PREREGISTRATION.md's Seen list has the Petrov reproduction, OpenAI's 20 rows recounted (including an o200k count for Italian on OpenAI's own sentence) and an unread cl100k smoke test. The page now says the predictions came before any count on the FLORES sentences from GPT-4o's vocabulary or the other 71, names what was already in hand, and keeps the helper's run (match tallies only; tokenizer-js/test-output.txt, mtime and ctime 17:12:41 UTC, before the 17:18:56 commit, a local record that is not in git). README.md matches.
- [fixed] MINOR-1, "Meta's Llama, Google's Gemma and Cohere's vocabularies need an account to download": Gemma 4 and Command A+ are not gated. The page now names the ten gated vocabularies and says those two were read from the originals, as tok/manifest.json records.
- [fixed] MINOR-2, "a pinned revision, listed in the table above" when the table showed none. page.json now carries each tokenizer's pin (file, or repository at revision, and the original a mirror copies), equal to tok/manifest.json for all 74, and the table prints it.
- [fixed] MINOR-3, "131 languages for which it reports a share of their own": Chinese and Kashmiri each appear in two scripts with one share between them. The sentence now says so.
- [fixed] MINOR-4, "the same count as an earlier one": MiniMax-Text-01 is matched to the later MiniMax-M2 by the dated-before-undated rule. Now "as another", and the rule and the case are on the page.
- [fixed] MINOR-5, the deviations the pre-registration promised to state on the page were not there, and DEVIATIONS.md item 4 said they were (and called a difference of up to 148 tokens "a few"). A paragraph in the check section now gives all of them, each figure recomputed by the verifier, and DEVIATIONS.md items 4 and 5 are corrected and added.
- [fixed] MINOR-6, four source-record URLs did not show the excerpted words (xAI pricing, Anthropic pricing, the python-genai repository root, xAI's API reference). Each now cites the page its kept copy came from; the online tool reads all four FOUND.
- [fixed] Found during the first pass and fixed before its report (014f0f3b): GPT-4's vocabulary was labelled 2023 although the inventory dates cl100k_base to 15 Dec 2022. Now 2022 everywhere, with "first shipped in December 2022 with an embedding model, three months before GPT-4 itself" and its three source entries, confirmed by the re-check.
- [fixed] NEW-1 (answers the one WRONG line), the verifier's figure match was a substring test, so a digit added in front ("142 of the 204", "11.7%", "113.70", og "174 tokenizers") or two figures swapped within one sentence (P3's top and middle thirds, Armenian and Georgian, Tamazight and Santali before and after, Persian, Italian, the GPT-4o fit) left it green, and some restatements ("those 73", "the other 71", "8 integers", the section 2 and check-panel counts) were never read. The verifier now matches a figure only as a whole number (no digit, and no comma or point followed by a digit, on either side), rebuilds every sentence that states more than one computed figure or restates a count from the recomputed values and requires it on the page word for word, and reads the count under the tiles. Every mutation the re-check reported, and 13 more restatements, now fail. A sweep that then changed the last digit of each of the 369 numbers in the page's text and meta copies, one at a time, caught 225; its survivors showed data-derived figures still unread (the six vocabulary buttons' years, Tekken from 2024, Qwen from 2023 to 2026, XLM-R's 2019 and mT5's 2020, the GPT-4o announcement date twice, the Anthropic tokenizer's year twice, "Ten vocabularies", the pre-registration's commit time, and "25.6 MB"), all now read (the last by --deep). A second sweep caught 272 of 369; every survivor is metadata, a section number, a digit in a name, the trainer's default, a quoted source's own number, or the archive size.
- [fixed] NEW-2, "Before looking at any count of our own on these sentences" is false of byte and code point totals, which the Seen list includes. Now "any token count of our own".
- [fixed] Found after publication, the same night (an unknown-token audit prompted by a research scout's note that Glot500 reports XLM-R "returns many UNK tokens"; neither round of this pass looked for unknown tokens, and the census's own round-trip audit had never run because tok/load.py's wrapper has no roundtrip method). XLM-R, the only one of the 74 tokenizers to emit any, puts one unknown token in place of letters it never learned, in 37 languages, so its counts there are what it emitted, not prices. The descriptive "best vocabulary for each language" now counts only vocabularies that encode every sentence in full (analyse.mjs, DEVIATIONS.md 6): all but twelve, not seven, pay under twice; median 1.38; the dearest is Tamasheq in Tifinagh at 4.16, not Shan at 3.28; mT5 95 and XLM-R 71, not 88 and 83. The page says so in a dated Corrections note, the price list marks the 37 cells, and the verifier recomputes both the corrected figures and the figures the note says the page first gave. P1 to P5 and "the fairest two" are unchanged under any treatment of the affected cells.