Artificial Wasteland · the razor, made operable
The Razor Is a Number
Everyone knows Occam's razor as a slogan: the simplest explanation is the correct one. Stated that way it is false — a good bet dressed up as a law. But there is a precise razor underneath the slogan, and it is a quantity you can watch get computed. Fit curves to noisy data below and three independent razors will all reject the wiggly curve that threads every point — in favour of a simpler one that never saw the truth.
Give a physicist noisy measurements and ask for the law behind them. A straight line misses the bends. A wild polynomial can be bent to pass exactly through every point — zero error, a perfect fit. Almost everyone's instinct says the perfect fit is worse, and almost no one can say why in a way you could hand to a machine. The razor is that "why," made arithmetic. Here it is running.
Fit the data. Find the razor.
A hidden true signal (faint amber) is sampled at a handful of points, each nudged by noise. You choose the polynomial degree; the fit is the least-squares curve of that degree. Drag degree up until the curve threads every dot — then read what it costs.
—
Push the degree to the top and the curve contorts to hit every dot: the train error collapses toward zero. But the true error — how far the fit strays from the hidden signal it was never shown — climbs. The perfect fit is memorising the noise. The razor's job is to find the degree where the curve has learned the signal and stopped there. Drag back down: somewhere in the middle both the true error bottoms out and the log-evidence peaks. That is the razor.
Why the perfect fit is the wrong fit
The same sweep, over every degree at once. Fitting the sample (blue) only ever improves. Predicting the truth (coral) gets better, then worse — the signature of over-fitting. The white line marks your current degree; the amber line marks where the razor cuts.
The razor is not a preference. It is a theorem.
You could dismiss all this as taste — we just like tidy curves. But there is a version of the razor that owes nothing to taste, and it falls straight out of the rules of probability. Ask not "which curve fits best?" but "which model — which whole family of curves of a given degree — made the data we actually saw most probable?" That number is the model's evidence, and computing it splits cleanly in two:
evidence = best fit it can manage × Occam factor
A more complex model can always fit at least as well, so the first term rises with degree. But a model flexible enough to explain anything must spread its bets across all the datasets it could have produced — and there is only so much probability to go around. Spread thin, it assigns little to the one dataset in front of you. That is the Occam factor, and it falls with degree. Nobody added a penalty for complexity by hand. Probability normalisation — the fact that the distribution must sum to one — imposed it automatically. The product peaks in the middle. [1]
The razor, decomposed
The log-evidence — the best fit plus the Occam factor — plotted against degree. It peaks exactly where the razor cuts. The two components that make it up trade off live in the tiles below; every value is computed in your browser.
Drag degree and watch the identity hold: the best fit climbs, the Occam factor sinks, and their sum — the evidence — turns over at the razor's degree. (Degrees 0–2 under-fit so badly they fall far below the window shown; the plot focuses on where the trade-off is decided.)
Three razors built on different foundations — the Bayesian evidence, Schwarz's BIC [3], and Akaike's AIC [2] — and the one thing you're normally not allowed to see, the error against the true signal. On rich data they land on the same neighbourhood. None of them ever picks the curve that threads every point.
The razor is a bet, not a proof
Press starve it above: six points, heavy noise. Now the razor chooses a simpler model than it did on rich data — often too simple to capture the real signal's bends. It has not failed. It has done exactly its job: cut to the complexity the data can justify, no more. When the evidence is thin, the honest inference is thin. The razor never promised to find the truth; it promised the best bet given what you were shown — and a bet on little evidence is a modest one. Simplicity here is a consequence of ignorance, not a claim about the world.
This is the crack the slogan papers over. "The simplest explanation is correct" would have the razor hand you truth. What it actually hands you is the best-supported guess, which shifts as the evidence does — richer data, and the same razor reaches for a more complex model without embarrassment. The instrument shows you both the razor's pick and the true error it's blind to, side by side, so you can watch them agree when data is plentiful and part ways when it is scarce.
Show the check
Every number above is recomputed in your browser from the same seeded data, and re-derived independently, offline, by research/occams-razor/verify.mjs (22 checks, all passing). The verifier confirms, without trusting the page:
- Training error falls monotonically with degree; the wiggliest fit threads the sample yet predicts the true signal worst.
- The log-evidence is computed two independent ways — a posterior-space form and an N×N kernel form — required to agree to one part in a million. (They do.)
- The Occam factor falls as the best fit rises, and their sum — the evidence — peaks at an interior degree. The maximum-evidence model is provably not the maximum-likelihood one.
- On the shipped data (sin 2πx, N=13, σ=0.15, seed 7) all three razors — evidence, BIC, AIC — and the true out-of-sample error agree on degree 5. Starve the data and the razor drops to a simpler model.
Run it yourself: node research/occams-razor/verify.mjs. The math follows Bishop's polynomial-fit treatment and MacKay's evidence decomposition; both are cited below.
What Ockham actually said (and four things everyone gets wrong)
The moat of this place is that it doesn't lie, and the razor's own history is a field of quiet, repeated small lies worth clearing.
The line every textbook quotes — Entia non sunt multiplicanda praeter necessitatem, "entities must not be multiplied beyond necessity" — appears nowhere in William of Ockham's surviving writing. It was documented as a myth by W. M. Thorburn in 1918; the earliest near-verbatim printed form is in the 1639 commentary of the Irish Franciscan John Punch, who already called it a common axiom — reporting a proverb, not coining one. [4]
What Ockham did write, repeatedly, was the principle in his own phrasings: "Numquam ponenda est pluralitas sine necessitate" — plurality is never to be posited without necessity (his commentary on the Sentences) — and "Frustra fit per plura quod potest fieri per pauciora" — it is futile to do with more what can be done with fewer (the Summa Logicae). [5] He used it as a working tool of theology and logic, not a law of nature.
And it was old when he reached for it. Ptolemy, in the Almagest, held it "a good principle to explain the phenomena by the simplest hypothesis possible." Aristotle preferred the demonstration "which derives from fewer postulates." Aquinas: "if a thing can be done adequately by means of one, it is superfluous to do it by means of several." Ockham's name stuck to a razor many hands had already stropped. [5]
The English term "Occam's razor" was coined only in 1852, by Sir William Hamilton, 9th Baronet — the Scottish metaphysician. Not William Rowan Hamilton, the Irish mathematician of quaternions fame; the two are constantly confused online (this page's own first search results confused them). A Latin "novacula Occami" had floated since Froidmont in 1649. [5]
"Everything should be made as simple as possible, but not simpler" is a paraphrase, not a verbatim Einstein quote. The sourced original, from his 1933 Herbert Spencer Lecture at Oxford, runs: the supreme goal of theory is "to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience." The pithy version is a later condensation. [6]
Even Newton's first Rule of Reasoning in the Principia is a razor in disguise: "We are to admit no more causes of natural things than such as are both true and sufficient to explain their appearances… Nature is pleased with simplicity, and affects not the pomp of superfluous causes." [5] A methodological rule — how to reason well — never a guarantee that nature is, in fact, simple.
The last honest catch: simplest has no fixed meaning
The instrument counted complexity in polynomial degree, and that felt neutral. It isn't. Change the description language and you change what counts as simple. Nelson Goodman's "grue" — green if examined before some future date, else blue — is, by its own lights, exactly as simple as "green"; the two are inter-definable, and only our chosen vocabulary makes one look natural and the other contrived. [7] The deepest formal notion of simplicity, Kolmogorov complexity, is provably defined only up to a constant that depends on which universal machine you describe things with. [8] There is no view from nowhere from which one hypothesis is simply, absolutely, simpler than another.
So the razor is real, and useful, and — as the philosopher Elliott Sober argues across a whole book called Ockham's Razors, plural — it has no single universal justification. Its edge is local: it comes from the model-selection or likelihood framework you commit to, the prior you choose, the language you count in. [9] That is not a weakness the instrument hides; it is the very thing the instrument shows. The razor cut at degree 5 because of a prior and a noise assumption you could have set differently — move them, and the cut moves. A tool whose answer depends on your assumptions is not thereby broken. It is just honest about being a tool.
The razor does not find the truth. It finds the simplest story your evidence can pay for — and tells you, if you let it, exactly how much it had to spend.