computing
The function-word diagnostic reattributes every held-out known essay in this edition before the fixed model scores the disputed set.
A public corpus, an old quarrel, a test that can fail
Twelve Papers, Thirty Little Words
Compare function-word rates in the disputed Federalist Papers. Held-out validation comes first, then a fixed model supports a probabilistic attribution toward Madison, not proof.
Hamilton and Madison wrote under the same name, on the same constitutional argument, within the same few months. Their large words had to follow the subject. Their small words did not. The useful evidence was the language neither man was trying to leave.
THE FEDERALIST PAPERS
No. 49
As the people are the only legitimate fountain of power, and it is from them that the constitutional charter, under which the several branches of government hold their power, is derived.
I. One word, used honestly
Put a disputed paper on the line
Start with upon. The bars are not published rates typed into a picture. They are corpus totals divided in your browser. Change the marker, then choose a paper.
Marker-word bench
Rates per 1,000 body-text tokens, Project Gutenberg #1404.
Disputed paper
Computing from the corpus
DIAGNOSTIC
With upon selected, Hamilton's known essays contain it hundreds of times. Madison's contain it seven times. The disputed set contains it three times across twelve essays. That is a striking clue, but a clue picked after looking can flatter itself.
You chose the cleanest word after seeing the answer. On twelve papers, that could be p-hacking with antique typography.
II. The method must go first
Hide every known name, one at a time
A test earns the disputed papers only after it faces papers whose authorship is already assigned. For each of 65 known Hamilton or Madison essays, the browser removes that essay, learns two author profiles from the rest, and predicts the hidden name. This is leave-one-out validation.
Held-out authorship test
Equal prior odds. Poisson word-count score. A fixed half-count smooths unseen words.
Correct names
computing
Balanced accuracy
computing
Average of Hamilton accuracy and Madison accuracy, so the larger Hamilton set cannot win by itself.
predicted Hamilton predicted Madison inset border means wrong
HELD OUT
The content-word switch is the control. war, executive, and legislature sound more informative because they carry meaning. Here they are worse than the majority-class shortcut once the unequal author counts are exposed. Their balanced accuracy is near a coin flip.
The 32-word set is not a reconstruction of Mosteller and Wallace's main model. It is a smaller, inspectable diagnostic built for this page. Its job is falsifiability: if tiny habitual words cannot recover the names already known, it has no standing to speak about the disputed names.
Leave-one-out validation alone does not rule out feature fishing. The four historical markers upon, while, whilst, and to are the classic Mosteller-Wallace set, chosen a priori rather than fished from this corpus; the 32-word page model remains a separate diagnostic.
computing
The three content words produce coin-flip balanced accuracy. Subject matter is not the clean separator here.
computing
Only after validation does the 32-word diagnostic score the reserved disputed set. No. 55 alone falls on Hamilton's side.
III. What the historical analysis actually established
Not one word, and not proof
Frederick Mosteller and David Wallace did not decide the case by drawing a line between two upon rates. Their 1963 JASA study compared discrimination methods and pooled about thirty marker words in Bayesian analyses that allowed word counts to vary more than a simple Poisson model permits. The book-length treatment followed in 1964.
Their result favored Madison for all twelve disputed papers. The strength was not uniform. The modern Founders Online editorial note, citing their book at page 263, gives the weakest result as 90 to 1 for No. 55, then 800 to 1 for No. 56. Other specifications and popular summaries give much larger figures. Those odds depend on the chosen words, model, and prior odds, so this page does not turn one dramatic figure into a universal answer.
That is probabilistic attribution, supported by historical evidence and later replications. It is not a recovered signature, an extant manuscript, or certainty. Douglass Adair had already noticed the contrast between Hamilton's while and Madison's whilst. Prior art supplied a tell. Mosteller and Wallace supplied a pooled statistical case.
The check
Observed data
Computing token totals and rates from 85 paper rows.
Diagnostic inference
Running 65 held-out classifications and scoring 12 reserved papers.
Published inference
Mosteller and Wallace's negative-binomial Bayesian odds are cited, not regenerated. No. 55 is explicitly kept as the weakest case.
Preparing the live arithmetic.
Free choices and uncertainty: body text only; ASCII letter tokens with internal apostrophes; 51 Hamilton and 14 Madison papers treated as known; Jay's five and the three joint papers excluded; equal author priors in the page diagnostic; a fixed 32-word list; independent Poisson scores with a 0.5 pseudo-count; no correction for dependence among words. Edition and tokenization move exact rates. The paper-level counts are observed data. The nearest-rate label and classifier are diagnostic inferences. Historical odds are published evidence. Run node research/federalist-stylometry/verify-federalist-stylometry.mjs.
Corpus convention and why our upon rate is 3.34, not 3.24
The parser splits Project Gutenberg eBook #1404 at its 85 Federalist headings, takes text after the salutation ending in either a colon or period and before PUBLIUS when that signature is present, lowercases it, and counts ASCII letter sequences with an optional internal apostrophe. On that declared convention, the known Hamilton corpus has 111,305 tokens and 372 uses of upon, or 3.342 per thousand. The known Madison corpus has 38,764 tokens and 7 uses, or 0.181 per thousand. The ratio is 18.5, somewhat larger than the often-published 3.24 versus 0.23 comparison. Different editions, author sets, inclusion of headings or notes, and token rules explain small shifts. We show our result rather than forcing the corpus onto the famous number.
Exactly what the page classifier is, and is not
For each word and author, it estimates a rate from all eligible training papers except the held-out one. A half-count prevents an unseen word from producing an infinite log score. For a paper of length N and count x, the word contributes x log(rate) - N rate; terms identical for both authors cancel. Contributions are summed with equal prior odds. This is a transparent Poisson naive Bayes score. It is not Mosteller and Wallace's negative-binomial model, does not reproduce their posterior odds, and treats correlated word counts as if independent.