A stratum · what the survivors can't tell you

The Planes That Didn't Come Back

The planes that came home were full of holes, and the obvious place to add protection was where the holes were. That is exactly wrong. The holes are a map of where a plane can be hit and still come home. The danger is where the survivors are unmarked. Below, run the idea, then run the 1943 memoranda it comes from, on their own numbers.

The mathematician Abraham Wald, working for the Statistical Research Group at Columbia, saw the trap. Every plane the brass could study had one thing in common: it came back. The planes hit in the truly fatal spots weren't in the data, because they never came back to be counted. So the survivors' damage is survival-filtered: holes pile up where damage is survivable, and thin out exactly where it kills. The empty patches on the returning planes mark the lethal regions. This is survivorship bias, and below you can run it.

Fly the fleet

Each region has a hidden lethality — the chance a single hit there downs the plane. Drag it up and watch two things: the plane is lost more often, and the holes on the planes that do return drain away from that region. You only ever see the survivors' holes (amber). The blue is what you're not shown: where the flak actually lands. Wald's move is to compare them.

Why the gaps, not the holes

Hits scatter across the plane in proportion to each region's exposed area — call that share aᵢ. A hit in region i downs the plane with probability vᵢ (its lethality). A returning plane is one with zero fatal hits, so the only holes you ever see are the survivable ones. Work it through and the survivor holes land in region i with probability

qᵢ ∝ aᵢ · (1 − vᵢ)

That is the whole bias in one line. Holes-per-area, qᵢ ⁄ aᵢ, is proportional to (1 − vᵢ): the more lethal a region, the fewer survivor holes it shows relative to its size. The naïve eye reads qᵢ and armors the biggest pile of holes — the fuselage, which is huge and rarely fatal. Wald reads qᵢ ⁄ aᵢ and armors the region the survivors are quietly missing — the engine.

And the one input the naïve reading throws away is aᵢ, the exposure baseline — together with the planes that never returned to be counted. Keep them, and you can do better than rank the regions: you can recover every lethality exactly. The fraction lost fixes the overall scale (Z = 1 + ln P(survive) ⁄ λ), and then vᵢ = 1 − qᵢ · Z ⁄ aᵢ reads each region's true danger straight off the survivors' damage. The instrument above does this live, in the red bars.

What Wald actually wrote

Everything above is a toy built on this page: five regions, lethalities we chose, hits we let fall at random. Wald's own work survives, and it can be run. It is eight memoranda he wrote for the Statistical Research Group at Columbia in 1943, reprinted in July 1980 by the Center for Naval Analyses as Research Contribution 432, a 101-page scan anyone can read. His examples are made up and he says so, closing one with “for the observed data of this hypothetical example”. But they are his numbers, worked by his method. Part V and Table 8 of Part VIII are recomputed below by the same file this page runs in your browser. The two Part I examples run only in the downloaded check; the examples in Parts III, IV and VI are not recomputed here.

The example the famous story is really about is in Part V. He sends 400 planes. 359 come home. He counts the holes on the ones that came home, part by part, and he knows how big each part is. That is all he has, and it is enough.

Read the fuselage row. It carries more holes than any other part, 78 of the 202, and it comes out as one of the safest places on the plane to be hit: a hit there downs it about one time in twenty. The engines carry only 39 holes while making up more than a quarter of the area the enemy fires at, and they come out worst, nearly two in five. That is the whole bias, in his own table, and the formula he uses, q(i) = δ(i) ⁄ γ(i) · q₀, is the one this page wrote as qᵢ ⁄ aᵢ ∝ (1 − vᵢ): share of the holes over share of the area.

What the legend leaves out

The story says look where the holes aren't. The memorandum needs three more things: measurements and an assumption.

• How big each part is, from where the shooting comes. Without the areas there is nothing to compare the holes against, and “fewest holes” just means “smallest part”. His footnote is specific: “By area is meant here the component of the area perpendicular to the direction of the enemy attack.”
• How many planes did not come back, and how many holes each returner has. This is how he gets q₀, the chance a plane survives one hit, without assuming anything about how hits are scattered across a fleet. 240 came home clean, 68 with one hole, down to 10 with four; 41 never came home. If one hit leaves a plane flying with chance q, the 68 one-hole planes are the survivors of 68 ⁄ q planes that ran into exactly one hit, the 29 of 29 ⁄ q² that ran into two, and so on up; all of those, with the 240 that ran into none, must make the 400 that went. That is the equation the instrument solves.
• That a hit's effect does not depend on the hits before it. He states it as an assumption, then spends Parts II to IV dropping it and bounding what can still be said, and Part VI on what a small mission's sampling error does to the answer.

None of this is Poisson and none of it is a slogan. It is a method that works exactly under its stated assumptions: a fleet flown in the check under those rules, with holes per plane drawn from a deliberately lopsided distribution, gives back every part's true survival chance to within sampling error.

The one place he writes the word

Part VIII makes the problem harder: three kinds of gun, flak, a 20-mm cannon and a 7.9-mm machine gun, each hitting the plane from its own usual angle, so the chance a hit lands on the engine depends on who fired it. His hypothetical mission sends 1,000 planes, 634 reach the target, 32 are lost, and the 602 that return carry 399 holes, tallied by gun and by part. His Table 8, recomputed here:

Recomputed live from his Table 7 counts and his hit-location probabilities. Each cell shows our figure; where his printed figure differs, it follows in brackets. The differences are all in the third decimal and come from his rounding each ratio to three places before multiplying.

And then, on page 89, the sentence the whole legend hangs from, and the only place the reprint's text layer turns up the word:

“These, and other conclusions that can be made from the table of vulnerabilities derived by the method of analysis of part VIII, can be used as guides for locating protective armor and can be used to make a prediction of the estimated loss of a future mission.”

So the armor is in the memoranda, but as a use for a table of probabilities, one line long, at the end of the hardest example. What he built was an estimator: from the damage on the survivors, a number for every part.

What recomputing it turned up

Three small arithmetic slips and one assumed baseline, none of which changes a conclusion he drew. Recomputing also means showing the ones that did not come out the same.

• Part V's q₀ is printed .850. The root of his own printed equation, .4q⁴ − .170q³ − .072q² − .030q − .025 = 0, is .8459 (at .850 the left side is +.0019, not zero), and with the unrounded 29 ⁄ 400 = .0725 it is .8467. Both are .85 to two places. The three other roots checked here, .851, .87 and .930, come out exactly as printed.
• 31 ⁄ 202 is .1535, printed .154. The rounding happens to make the fuel system's share of the holes equal its share of the area, so its q equals q₀ exactly. From the raw counts the fuselage is .944 and the fuel system .845, where he prints .95 and .85.
• A bracket in Part VIII sums to 1.0435, printed 1.0433, which moves λ from .9704 to his .9703. Nothing downstream notices.
• Table 8's last row rests on areas the footnote says it is assuming, .084, .128, .212 and .576. The hit mix implied by Part VIII's own numbers gives .082, .136, .210 and .571 instead, and with those the engine's vulnerability to an unspecified hit is .229 rather than .179. The engine is still the worst part. But the gap is the page's whole point in miniature: move the baseline a little and the answer moves, because the baseline is half the answer.

Where the story comes from

The memoranda were wartime work, and in the words of the 1980 reprint's cover letter “never published externally” in the 37 years between. The version everyone knows was assembled later, and you can watch it gain details the sources do not have.

• 1945. The Statistical Research Group's final report, quoted by its director W. Allen Wallis in 1980, says aircraft vulnerability led the group to devise “a technique for determining vulnerability from damage survived by our own planes”.
• 1980. An eyewitness telling of the anecdote appears in Wallis's reply to the comments on his history of the group (J. Amer. Statist. Assoc. 75, 334–335), where he also says the work “has never seen the light of day”: “The military was inclined to provide protection for those parts that on returning planes showed the most hits.” Wald, he goes on, assumed hits were spread evenly over the plane, and “from these premises, he devised methods for estimating the vulnerability of various parts.” No service is named, no aircraft, no theatre, and the punchline is a method, not a location. The same year the Center for Naval Analyses reprinted the memoranda. (We read this reply as quoted at length by Timothy Taylor, 2020; the journal page itself is behind JSTOR.)
• 1984. Marc Mangel and Francisco Samaniego publish the first reconstruction of the method in the same journal. Their article's abstract says the work was used in World War II and in Korea and Vietnam. Asked in their rejoinder whether it was used, they write: “We do not know whether it was used during World War II, although it was produced early enough in the war to have been available.” What they can document is later: in Vietnam, analysts at the Center for Naval Analyses used it on the A-4, and “their analysis led to structural modifications that improved the A-4's survivability.”
• 2008. John D. Cook's post of 21 January already puts British bombers returning to England in the story, without naming the RAF. The wording survives in a 1 March 2010 archive.
• 2010. A blog post at Mother Jones names the RAF: “Back during World War II, the RAF lost a lot of planes to German anti-aircraft fire. So they decided to armor them up.”
• 2016. The red-dotted bomber that illustrates the idea everywhere is uploaded to Wikimedia Commons on 12 November 2016, described by its maker as an “illustration of hypothetical damage pattern on a WW2 bomber”, its dots “roughly based on” the Mother Jones picture. The aircraft drawn under the dots is a Lockheed PV-1 Ventura, a US Navy patrol bomber.

So the popular details, the Air Force asking (there was no US Air Force in 1943; the Army Air Forces became one in 1947), the bombers coming back from Europe, are not in anything Wald or Wallis wrote about it. The engines do come out worst in his Part V, but in an example he calls hypothetical. What is in the sources is better than the story: a man who could turn the holes on the survivors into a probability for every part of the plane, and a warning, in the form of an equation, about what happens when you forget the ones that did not come back.

The same hole in everything

Wald's bombers are the cleanest case, but the shape is everywhere a sample is filtered by the very outcome you're studying — you measure the survivors and forget the fallen:

• “College dropouts get rich.” You see Gates and Jobs; you don't see the millions of dropouts who didn't. The successful founders are the returning planes.
• “They built things better back then.” Buildings that depreciate fastest tend to be demolished first; the ones left to admire are a selected sample. Time is the armor test.
• Mutual funds with great ten-year records. The losers were quietly closed and merged away, so the surviving funds' average overstates the field — the dead funds are off the leaderboard.
• “This treatment works — look how well our patients did.” If the sickest dropped out, the survivors flatter the cure. Selection on the outcome, again.

In each one the fix is Wald's: ask what got filtered out of the sample by the thing I'm trying to measure, and put it back. The gap in the data is the data.

▸ part of the portal: The Condition You Weren't Told