The Verification Venue · a method that fails harder the more you feed it

More Data, More Certain, Still Wrong

Here is a four-species tree. We know it is the truth, because we are about to grow the DNA on it ourselves. Then we hand the DNA to a reconstruction method and ask for the tree back. In one region of branch-length space the method returns a tree that never existed, and returns it harder the more sequence you give it.

Every phylogeny you have ever seen is an inference. Nobody watched the branchings. What makes the inference trustworthy is supposed to be a guarantee called statistical consistency: as the data grow without bound, the estimate converges on the truth. In 1978 Joe Felsenstein showed that maximum parsimony, then the dominant method, does not have that guarantee. Worse than not having it: there are trees for which parsimony converges on a specific wrong answer, with the support for that wrong answer growing steadily as you add sites.

The bench below is that proof, made operable. The tree on the left is real by construction. Drag its branch lengths. The verdict panel computes, exactly and instantly, what parsimony would say if you handed it infinitely many sites.

The map below is the whole parameter plane, coloured by that same exact calculation: which topology wins the informative-pattern count when the data are unlimited. The pale region is safe. The dark region is what the literature calls the Felsenstein zone. Your current settings are the ring.

Now grow the data

The exact verdict says what happens at infinity. Here is what happens on the way there. For each sequence length we simulate whole alignments down the true tree under Jukes-Cantor, score all three topologies by Fitch parsimony, and record how often the true one wins outright. The blue curve is a safe tree (long branches 0.25). The red curve is the same tree with one number changed: the long branches lengthened to 0.45.

Read the red curve again, slowly. It is not noisy, and it is not flat. It goes down. A hundredfold increase in evidence takes the method from occasionally right to reliably wrong, and it would keep going: the wrong topology's informative-pattern probability exceeds the true one's by a fixed margin, so the error is a law of large numbers working perfectly, on the wrong quantity. Bootstrap support for the wrong tree climbs toward 100% for exactly the same reason, resampling the sites cannot escape what the sites say. That last point is not computed on this bench; it is a known result, shown by simulation in Hillis & Bull, Syst. Biol. 42:182-192 (1993), and it follows from the bootstrap proportion tracking the same asymptotic winner. Confidence measures repeatability, not truth.

The mechanism has a name, long-branch attraction. Two lineages that have each changed a great deal will, by chance alone, sometimes land on the same base at the same site. Parsimony reads shared bases as shared ancestry, and cannot tell a coincidence from an inheritance. The two long branches are given many chances to coincide; the short internal branch, which carries the real signal, is given almost none.

This is not only a simulation result. In the 1990s microsporidia (fast-evolving parasites with reduced genomes) were placed at the base of the eukaryote tree by early rRNA analyses, and were later reassigned to the fungi once the long-branch artefact was recognised and denser taxon sampling and better models were applied. The Ecdysozoa hypothesis, that arthropods and nematodes share a moulting clade, was resisted in part because the fast-evolving nematode Caenorhabditis was being pulled toward other long branches. Both cases are now settled; they are here as history, not as open questions.

The answer everyone gives

If you have taken a phylogenetics course you already know the reply, and it is correct: parsimony is naive, use maximum likelihood, which is consistent. So let us do that, on exactly the data that just broke parsimony. Same simulation, same zone, but the three topologies are now scored by fitting Jukes-Cantor branch lengths and comparing log-likelihoods.

Likelihood wins, decisively, and it wins because more data helps it. That is consistency, demonstrated rather than asserted. The lecture could end here, and usually does.

Where consistency actually lives

It should not end here, because the guarantee has a precondition that gets dropped in the retelling. Likelihood is consistent when the model contains the truth. The theorem is about the pair (method, model), and the word "consistent" attaches to the pair. Take the truth out of the model and the guarantee goes with it.

So here is a process the Jukes-Cantor tree model does not contain. Real sites do not all evolve at the same relative rate down every lineage; the pattern changes over time, a phenomenon called heterotachy. We build the simplest possible version: the true topology stays AB|CD throughout, but a fraction ρ of sites evolve with A and C on long branches, and the remaining sites evolve with B and D on the long branches instead. Every site is generated on the true tree. Nothing is faked. The alignment simply has no single set of branch lengths that describes all of it.

Three methods now compete on that alignment: parsimony, likelihood fitting one homogeneous Jukes-Cantor tree (the misspecified model), and likelihood fitting a two-class mixture of branch lengths with the class weight free (a model that does contain the generating process). Slide ρ and run.

At ρ near 0 or 1 there is no heterotachy, the homogeneous model is correct, and likelihood is far ahead. As ρ approaches 0.5 the misspecification bites, and the blue curve falls below the green one: on this bench, at that setting, parsimony beats likelihood. Parsimony did not improve. Its curve is nearly flat across the whole range: it is not good here, it is merely unbothered, because it never made a claim about the process in the first place. What changed is that likelihood's claim became false.

And the purple curve is the point. Give likelihood a model that contains the heterogeneity and it is back on top wherever the heterogeneity is real, without changing the method at all. Read the endpoints honestly, though: at ρ = 0 and ρ = 1 there is no heterotachy to model, and the mixture pays for its extra parameters, finishing 0.0833 and 0.0333 below plain Jukes-Cantor likelihood on this bench. That is the ordinary cost of fitting a richer model than the data needs, and it is small next to the 0.8833 the plain model loses at ρ = 0.50 by being wrong. Consistency was never a property of "likelihood". It was a property of a model that happened to be right.

The part that is still an argument

That green-beats-blue crossing is not a curiosity we invented. It is the subject of a sharp published exchange, and the exchange is worth laying out because the disagreement is about exactly the quantity you were just sliding.

Evolutionary rates vary among sites and across the phylogenetic tree (heterotachy). A recent analysis suggested that parsimony can be better than standard likelihood at recovering the true tree given heterotachy. Spencer, Susko & Roger, Mol. Biol. Evol. 22:1161 (2005), opening lines

• The claim. Kolaczkowski & Thornton, Nature 431:980-984 (2004), simulated four-taxon alignments concatenated from two different branch-length regimes and reported that maximum parsimony generally outperformed maximum likelihood and Bayesian methods across much of the parameter space they examined. Their abstract puts it as "Maximum parsimony performs substantially better than current parametric methods over a wide range of conditions tested", and that phrasing is the whole fight: the claim is bounded by the conditions tested, and every rebuttal below is an argument about whether those conditions are the ones nature hands you. The claim itself, that there exist regimes where parsimony wins, is not in dispute, and your slider finds one.
• The model rebuttal. Spencer, Susko & Roger, MBE 22:1161 (2005), argued the result is limited to a special case, that the deterministic assignment of the first half of sites to one regime and the second half to the other is not the right statistical model of the situation. Their abstract states that the earlier mixture model "was inconsistent because it was incorrectly implemented", and their answer is a correctly implemented mixture. That is what the purple curve is. We could not read the paper's body text (paywalled), so we do not quote any figure from it here.
• The parameter-range rebuttal. Gadagkar & Kumar, MBE 22:2139 (2005), re-ran the comparison over the full range of proportions of sites involved in heterotachy rather than the single 50:50 split, and concluded that likelihood is significantly superior overall even under heterotachy. That is what happens to the blue curve as you slide ρ away from 0.5.
• The design rebuttal. Philippe and colleagues, BMC Evol. Biol. 5:50 (2005), argued that the specific design simultaneously changed the level of heterotachy and the average terminal branch length, so the two effects are not separable in the published results, and that keeping all terminal branches equal is biologically unrealistic.

We are not adjudicating this. Our bench reproduces the shape of the disagreement rather than any paper's exact parameter grid: our regime assignment is random per site with probability ρ, not a fixed first-half/second-half split, and our branch-length values are our own. What the bench does establish, and what nobody in the exchange disputes, is the structural claim: whether parsimony or likelihood is more accurate here is a function of a nuisance parameter nobody can measure directly, and both sides were arguing about where in that parameter's range real data live. Slide ρ and you are holding the axis of the argument in your hand.