The specimen is yours · language
The Encoding Your Text File Admits To
Drop a text file. Your browser will prove which Unicode forms its complete bytes satisfy, show the surviving readings, and keep a statistical favorite separate from what the author actually meant.
A byte string can be accepted by more than one encoding and say different things under each. That is not an algorithm failing to think hard enough. It is missing evidence. For a fixed list of encodings every computation here terminates; what the bytes do not contain is a label saying which reading was intended.
Does the complete byte string obey a named encoding grammar?
Which candidate looks most like the detector's learned patterns?
Which encoding did the writer or program mean? Bytes alone may not say.
Published anchor · runs before your file
Unicode Table 3-7, reproduced first
Unicode 17.0 prints three test vectors beside its nine legal UTF-8 forms. The independent state machine used on your file and the browser's fatal UTF-8 decoder both run them now.
| printed bytes | standard says | Table 3-7 machine | fatal TextDecoder | result |
|---|
Nine boundary probes, derived from the rows rather than printed beside them
Three printed vectors cannot separate a correct Table 3-7 from a wrong one. The offline verifier corrupts each row six ways a person actually gets it wrong and shows that a pass or fail verdict on the printed vectors alone catches only two of the six. These probes each sit one byte outside a single row limit, which is what closes the gap. They are derived here, not quoted, and are labelled as such.
| bytes | which limit it probes | derived expectation | Table 3-7 machine | result |
|---|
The four printed U+FFFD tables, which discriminate where a boolean cannot
Unicode 17.0 section 3.9.6 prints Tables 3-8 to 3-11: what a replacing decoder should emit for ill-formed input. This page reproduces them with its own maximal-subpart routine, which never calls TextDecoder, and with the browser's replacement decoder. Table 3-11 is the discriminating row: nine bytes give four U+FFFD under maximal-subpart substitution and eight under naive per-byte substitution, so it separates two rules that a reject-or-accept test reports identically. The standard calls this substitution practice recommended and explicitly not required for conformance; the normative neighbour is the weaker rule in 3.9.5 that a converter must not consume bytes that begin a well-formed sequence.
| printed table | input bytes | printed output | local maximal-subpart routine | result |
|---|
Running the published vectors.
Anchor: The Unicode Standard, Version 17.0, Chapter 3, definition D92 and Table 3-7, with Tables 3-8 to 3-11 from section 3.9.6. The printed outcomes are C0 AF ill-formed, E0 9F 80 ill-formed, and F4 80 83 92 well-formed. RFC 3629 agrees byte for byte and says so without claiming authority: its section 4 states that the authoritative definition is in the Unicode Standard and that its own grammar is believed to describe the same thing.
Layer one · your complete file
Put the bytes on the bench
TXT, CSV, TSV, Markdown, JSON, XML, subtitles and logs are all welcome. Binary input is allowed too, but a valid encoding grammar does not prove the bytes are text.
A ceiling, added after measurement rather than designed in. Review of this page found that twelve simultaneous decodings with a tracked source span per character cost about 350 bytes of browser memory per input byte. Headless Chromium measurements: 262,144 bytes cost 109 MB and 263 ms, 1,048,576 bytes cost 349 MB and 1,029 ms, 8,388,608 bytes cost 2,510 MB and 7,686 ms, and 16,777,216 bytes crashed the renderer outright, which is a failure no code on this page could have caught or explained. So a file larger than 262,144 bytes is now refused before a single byte is read, and the refusal prints the limit and the file's own size. Sampling a prefix instead would silently answer a different question, and the whole point of the instrument is that it reads all of the file or none of it.
Waiting for the published anchor.
The page's inline Content Security Policy says connect-src 'none'; form-action 'none'; worker-src 'none'. The file is read by File.arrayBuffer() into page memory. There is no upload function. Open your browser's Network panel before choosing a file if you want the browser's own account.
specimen
What the bytes establish
Longest signatures are checked first. A BOM is shown and hashed, never silently stripped. The three bytes EF BB BF on their own are accepted by 8 of this page's 12 candidates and produce 7 distinct readings, so a BOM is a strong convention rather than a fact in the bytes.
The validator tracks a source span for every Unicode scalar. On refusal it names the first byte that cannot occupy the required range. A zero-based offset is for code; the one-based offset is for a hex editor.
A start-of-file EF BB BF is strong evidence for UTF-8, but Unicode calls it a near-certain indication rather than proof. UTF-8 does not require it.
Rows matched by this file
Counted across their ranges, the nine rows admit exactly 1,112,064 distinct characters and leave thirteen byte values, C0, C1 and F5 to FF, that cannot appear anywhere in a well-formed file. That count is derived here, not quoted: chapter 2 prints 1,114,112 code points in the codespace and 2,048 surrogate code points, and the rows encode the difference.
| row | pattern | scalars |
|---|
Strict readings, not guesses
Every reading this instrument can audit byte by byte
The finite set is UTF-8, both UTF-16 byte orders, both UTF-32 byte orders, and seven shipped WHATWG single-byte indexes. Strict means an odd UTF-16 length, a lone surrogate, an out-of-range UTF-32 value, a null single-byte index entry, or an illegal UTF-8 byte refuses at an exact offset. No string is normalized to NFC or NFD before comparison.
The lenient UTF-8 preview is a separate object
Replacement decoder output contains 0 U+FFFD replacement characters. A replacement is the decoder handling an error. It does not show that U+FFFD was present in the source.
Browser label trap. In the Web Encoding Standard, the labels ascii, latin1, iso-8859-1, us-ascii and cp1252 all resolve to windows-1252. This page displays one row for that browser-standard reading. Software outside the Web may use those labels differently.
A guess is allowed, under its own heading
What chardet 2.2.0 favors
Now, and only now, the statistical detector sees the same complete Uint8Array. Its confidence is an implementation score, not a calibrated probability. Short files, mixed languages, source code, tables, identifiers and binary data are poor terrain for language-frequency models.
Full-file ranking
| rank | candidate | opaque score |
|---|
Recognizer roster in the vendored source
There are 27 recognizers. Five ISO recognizers choose an ISO label or a Windows label according to whether C1 bytes occur, so each conditional pair above is one recognizer, not two simultaneous results. The package also contains windows-1257, windows-1258, windows-874 and ASCII recognizers. A detector name does not add that encoding to the strict preview set above.
The scouting specification listed 30 labels and omitted those four recognizers. The vendored 2.2.0 entry point constructs 27 recognizer objects and makes five names conditional. This page follows the executable source rather than preserving the scouting list.
Layer two · answer the detector dismissal
The equivalence grid
A detector can guess. This grid does something less glamorous and more final. Every pair of strict candidates is compared over the complete file. Matching means identical Unicode scalar sequences with no normalization. A difference reports the first scalar and both candidates' exact source-byte spans.
| candidate A | candidate B | relation | first evidence |
|---|
Which of these candidates could ever have said no
This is the sentence most encoding demonstrations get wrong, including the one this page opened with. "These bytes are also a valid windows-1252 string" sounds like a finding. It is not. Under the WHATWG index that browsers use, windows-1252 has an entry for all 256 byte values, so every byte string that has ever existed is a valid windows-1252 string: a JPEG, a random number, this sentence. An assertion that cannot be false is not evidence, and a check built on one is decoration. What carries information is not that the legacy reading succeeded but that it says something different, which is what the grid above reports and what the workbench below reconstructs.
| shipped table | coverage of the 256 byte values | undefined byte values | can a file be refused |
|---|
And which published table you read decides even that. The Microsoft vendor table at unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1252.TXT leaves undefined, five byte values that the WHATWG index maps to U+0081, U+008D, U+008F, U+0090 and U+009D. This page ships the WHATWG index, because it is describing what a browser does. Under the vendor table windows-1252 would not be total, and a file containing any of those five bytes would refuse instead of quietly producing text. The choice of table, not any property of your file, is what decides that.
The same trap runs through the usual framing. "UTF-8 versus ISO-8859-1" is, in any browser, UTF-8 versus windows-1252, because the Encoding Standard resolves that label there. So the familiar contrast is a real quantity compared against a constant. Failure is silent in exactly one direction: UTF-8 bytes decode as windows-1252 without raising anything and give different text, while legacy bytes usually refuse loudly as UTF-8. The check panel runs that asymmetry on constructed bytes rather than asserting it here.
An impossibility witness small enough to hold in one hand
The bytes for same ASCII bytes are fed to UTF-8 and all seven supported ASCII-compatible single-byte tables. Result: 8 candidates enter.
This is underdetermination, not undecidability. The finite comparison terminates. It returns several exactly equivalent hypotheses because the specimen contains no distinguishing byte.
The opposite overstatement is also worth refusing. The bytes are not uninformative: every refusal in the grid above eliminates a reading outright, and how much a file constrains its own encoding varies enormously with the encoding and the language. What the bytes cannot do is choose between the readings that survive.
The favorite has a history
Every power-of-two prefix, then the whole file
A single confidence score hides whether the answer held still while evidence arrived. The curve reruns chardet at 1, 2, 4, 8 bytes and onward through the greatest power of two below the file length, then once on the full file. Ties are printed, not silently broken. Prefixes diagnose stability and never replace the complete-file result.
| prefix bytes | top | score | top three |
|---|
A repair has to make the round trip
The mojibake workbench
The common damage chain is reversible when the wrong single-byte reading can be encoded back into the same bytes and those bytes form strict UTF-8. Choose the intermediate single-byte table. The workbench prints every operation. It refuses if any mapping is undefined, never normalizes, never alters the original, and never offers a repaired download.
Running the chain.
A byte-equality certificate proves that a proposed transformation is reversible. It does not prove the repaired wording was intended. Human language knowledge and provenance still matter.
The check
Two implementations, complete bytes, visible refusals
Waiting for the specimen checks.
Fixed versions and set sizes.
Every free choice and uncertainty.
- The main standards verdict and SHA-256 use the complete byte array. There is no sampling. Prefix lengths are powers of two plus full length and are diagnostic only.
- UTF-8 is checked by an explicit nine-row state machine and independently by TextDecoder('utf-8', {fatal:true, ignoreBOM:true}). Every other browser-supported strict label is also cross-checked with a fatal TextDecoder. UTF-16 and UTF-32 source spans are tracked locally because TextDecoder does not report offsets and has no UTF-32 label. Seven single-byte indexes are shipped with the page.
- BOM signatures are checked longest first and shown without removal. Hashes always include them. Display is BOM-preserving.
- Strict failure and lenient replacement are separate. Control characters are escaped in previews. Preview length is capped at 1,200 scalars, while validation and comparison use the complete decoding.
- Scalar sequences are compared without NFC or NFD normalization. First-index-pointer wins for reverse single-byte encoding. Undefined mappings refuse the chain.
- chardet scores are opaque ranks, not probabilities. Its 27 recognizers are printed from the vendored 2.2.0 source roster, including conditional ISO or Windows labels. Equal scores are labelled ties. Its models, mapping tables, scripts and fallback are same-origin.
- Five of the seven shipped single-byte tables have an entry for all 256 byte values, so their acceptance of any file is a constant and only their reading is informative. The two with undefined entries are the only ones that can refuse. The single-byte index is WHATWG's; the Microsoft vendor table for cp1252 differs on five byte values and would make windows-1252 refuse rather than accept them.
- Files larger than 262,144 bytes are refused before being read. That ceiling was added after review measured this page's memory cost, not chosen in advance, and the measurements behind it are printed above the drop zone. Everything under the ceiling is still read and analyzed in full, with no sampling.
- A zero-byte specimen is accepted by every candidate, so the page marks the equivalence grid vacuous and refuses to issue a byte-equality certificate over it rather than reporting a perfect agreement about nothing.
- The instrument audits a named finite set, not every historical encoding. A valid reading need not be meaningful text. A reversible repair need not be the author's text.
- Filenames, bytes, hashes, readings, ranks and interactions stay in page memory. The observer began after this module graph loaded, so the baseline count names earlier same-origin resources and the post-load count names anything that starts later.
Why the score stays in the second column
A published detector result, with its walls left standing
Pomikálek and Suchomel's 2011 chared paper used byte trigrams, about a thousand Web pages per language, labels taken from meta tags, and five-fold cross-validation with the language already known. It called the evaluation preliminary and did not compare against other systems. Those accuracies do not transfer to chardet 2.2.0 or to an arbitrary file dropped here.
Open the paper's complete Table 1, transcribed as printed
Each language has a Web frequency and a cross-validated accuracy. The paper prints n/a when its table gives no accuracy. Nothing in this transcription is a result from the detector running above.
| encoding | Czech | English | German | Greek | Italian | Norwegian | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| freq | accuracy | freq | accuracy | freq | accuracy | freq | accuracy | freq | accuracy | freq | accuracy | |
| utf-8 | 60.2% | 100.0% | 56.9% | 95.8% | 54.6% | 100.0% | 68.5% | 100.0% | 54.2% | 100.0% | 63.0% | 100.0% |
| windows-1250 | 32.2% | 100.0% | 0.3% | n/a | 0.1% | n/a | 0.2% | n/a | 0.0% | n/a | 0.1% | n/a |
| windows-1252 | 0.4% | n/a | 9.4% | 97.5% | 6.5% | 97.3% | 3.1% | 75.8% | 7.1% | 95.7% | 7.0% | 97.4% |
| windows-1253 | 0.0% | n/a | 0.0% | n/a | 0.0% | n/a | 14.3% | 99.3% | 0.0% | n/a | 0.0% | n/a |
| iso-8859-1 | 1.0% | 89.5% | 32.8% | 90.9% | 37.1% | 85.8% | 1.7% | 71.2% | 37.9% | 85.1% | 29.3% | 88.2% |
| iso-8859-2 | 6.0% | 99.6% | 0.0% | n/a | 0.1% | n/a | 0.0% | n/a | 0.1% | n/a | 0.1% | n/a |
| iso-8859-7 | 0.0% | n/a | 0.0% | n/a | 0.0% | n/a | 12.0% | 97.2% | 0.0% | n/a | 0.0% | n/a |
| iso-8859-15 | 0.0% | n/a | 0.0% | n/a | 1.2% | 85.6% | 0.0% | n/a | 0.0% | n/a | 0.4% | n/a |
| training docs | 801 | 668 | 773 | 879 | 771 | 740 | ||||||
| w. avg accuracy | 99.2% | 93.5% | 93.7% | 97.9% | 93.3% | 95.7% | ||||||
The next table isolates the two summary rows and adds one derived column. Error rate is not printed by the paper. It is calculated here as 100 minus weighted average accuracy.
| language | training documents | weighted average accuracy | derived error rate |
|---|---|---|---|
| Czech | 801 | 99.2% | 0.8% |
| English | 668 | 93.5% | 6.5% |
| German | 773 | 93.7% | 6.3% |
| Greek | 879 | 97.9% | 2.1% |
| Italian | 771 | 93.3% | 6.7% |
| Norwegian | 740 | 95.7% | 4.3% |
Selected low cells from the printed table expose the confusions: ISO-8859-1 was 71.2% for Greek, 85.1% for Italian, and 85.8% for German. The error rates above are derived here as 100 minus the paper's printed weighted average.
The moat is inspectable
Nothing leaves the device
The policy in this document's head blocks connections, forms, workers, frames, objects and media. The instrument contains no fetch, XMLHttpRequest, WebSocket, beacon, analytics or storage call. Font CSS, scripts and icons are same-origin. The byte array exists only in this page's memory until the page closes or another file replaces it.
Resource names observed after instrument load:
none
A zero here is the page's own report. The stronger check is yours: open Developer Tools, select Network, choose a file, and watch that no request carries its name, bytes, hash or text.
What this does not prove
The missing label stays missing
- It does not prove which program wrote the file or which encoding its author selected.
- It does not prove what language the file contains, that any surviving reading is meaningful, or that the bytes are textual.
- It does not prove a mojibake repair recovers intent. Byte reversibility is narrower than authorship.
- It does not audit every historical encoding. Detector candidates outside the strict set are ranks without a byte-traced preview here.
- It does not treat Unicode normalization as character encoding. No normalization is performed.
- It cannot prevent the browser or operating system itself from having defects. The independent state machine and published vectors narrow that uncertainty but do not abolish it.