The Hours Between Best and Good

I / The claim, at the strength it was printed

Berlin: ten soloists, ten good violinists, ten future teachers

In 1993 K. Anders Ericsson, Ralf Th. Krampe and Clemens Tesch-Römer published in Psychological Review a study of violin students at the Music Academy of West Berlin. The professors nominated students with the potential for careers as international soloists; of 14 nominated, 10 took part, and the paper calls them the best violinists. From a larger pool of good students in the same department the authors chose 10, matched on sex and age, and a further 10 came from the academy's music-education department, which had lower admission standards. The paper calls that group the music teachers, because teaching was their most likely future. Each violinist estimated, for every year since they began, how many hours a week they had practised alone. The weekly estimates of practice alone can easily be converted to estimated yearly amounts by multiplication of the number of weeks in a year.

The paper's test was made at 18, and it says why: To avoid any confounding influences from the activities at the music academy, we statistically analyzed the amount of practice the young violinists had accumulated by age 18. Here is the result, in the authors' words.

At this age, the best young violinists had accumulated an average of 7,410 hr of practice, which is reliably different from 5,301 hr, the average number of hours accumulated by the good violinists, F(1, 27) = 4.59, p < .05. The average of the best two groups was reliably different from that of the music teachers, who had accumulated 3,420 hr of practice by age 18, F(1, 27) = 11.86, p < .01. Hence, there is complete correspondence between the skill level of the groups and their average accumulation of practice time alone with the violin.Ericsson, Krampe and Tesch-Römer (1993), Psychological Review 100(3), p. 379.

And the claim was larger than one comparison. The abstract of the same paper:

Individual differences, even among elite performers, are closely related to assessed amounts of deliberate practice. Many characteristics once believed to reflect innate talent are actually the result of intense practice extended for a minimum of 10 years.Ericsson, Krampe and Tesch-Römer (1993), abstract, p. 363.

II / Where the 10,000 comes from

Not from the sentence above

The test was made at 18, and the number was 7,410. The paper's Figure 9 draws accumulated practice on to age 20, and there the best group's curve ends close to 10,000 hours (read by eye from the printed figure; this page has not digitised it). In the prose, 10,000 hours appears once, in a passage about how the body adapts to training: the effects of over 10,000 h of deliberate practice extended over more than a decade (pp. 393 to 394).

Malcolm Gladwell's 2008 book Outliers made the number a rule. Ericsson and Harwell (2019) record that he called it the magic number for true expertise: ten thousand hours. Their answer: Although our research showed that an extended period of training and practice was required for attaining international-level performance, there was no evidence for a magical number. The number kept travelling anyway. When Ericsson died in June 2020, his university's notice said that he pioneered the concept that it takes 10,000 hours of practice to become an expert.

The replication reproduces the round number on its own terms: By age 20, both the best and good violinists had accumulated more than 10 000 h of practice alone on average. Recomputed from the released histories, the best group averaged 10,586 hours by 20, all 13 histories complete. The good group's 11,804 counts 7 histories that stop before 20, so it is a floor; the 6 good violinists who had reached 20 averaged 9,191.

This page does not grade the 10,000-hour rule. It grades a narrower sentence, and the sentence is the one the 1993 authors wrote.

III / The replication, with its rows in the open

Cleveland: the same design, run blind

Brooke N. Macnamara and Megha Maitra ran the study again at the Cleveland Institute of Music. Faculty nominated 24 students with the potential for careers as international soloists, and 13 agreed to take part: the best violinists. 13 good violinists were matched to them on sex and age as far as possible, and 13 violin students from the music department of neighbouring Case Western Reserve University formed the least accomplished group. Recruitment ran from the summer of 2014 to the autumn of 2018. The analysis was preregistered in April 2017, before the first author had seen the data: The first author did not collect data from any of the student violinists and has not coded or looked at the data.

The interviews were double-blind: interviewers did not know a violinist's group, and violinists were not told there were groups. The replication's authors observe: There is no indication in Ericsson et al. that experimenters were blind to the participants' skill level. This page reports that as a difference of design and nothing more. Here is what the replication found, in its authors' words.

We found a significant effect of group for accumulated practice alone until age 18, χ²₂ = 13.90, p = 0.001, η² = 0.26. However, the best violinists (M = 8224, 95% CI [6400, 10 048], range = 3978–14 664) had not accumulated significantly more practice alone by age 18 than the good violinists (M = 9844, 95% CI [6937, 12 751], range = 3120–21 268), t₂₄ = −0.93, p = 0.364, d = −0.38. The good violinists had accumulated significantly more practice alone by age 18 than the less accomplished violinists (M = 4558, 95% CI [3264, 5851], range = 2522–10 972), t₁₆.₅₇ = 3.26, p = 0.005, d = 1.33.Macnamara and Maitra (2019), Royal Society Open Science 6: 190327, p. 15. CC BY 4.0.

We did not replicate Ericsson et al.'s [1] major result of ‘complete correspondence between the skill level of the groups and their average accumulation of practice time alone with the violin’ (p. 379).Macnamara and Maitra (2019), p. 16.

At 18, with practice alone, every number the replication printed comes back from the weekly rows (the full list is in the check below): best 8,224 h, good 9,844 h, least accomplished 4,558 h; best against good Student t(24) = −0.93, p = 0.364; good against least accomplished Welch t(16.57) = 3.26, p = 0.005. And the replication's plainest sentence is true of its own rows: In fact, the majority of the best violinists had accumulated less practice alone than the average amount of the good violinists. Counted here: 8 of 13.

Two conventions matter for that match. The printed intervals are the mean plus or minus 1.96 standard errors (a t interval does not reproduce them), and the printed d is 2t divided by the square root of the degrees of freedom; with the pooled standard deviation instead, best against good is d = −0.36. The 2019 preregistration names the rule for choosing between the two t tests: if Levene's test were to be violated for a t-test, we would conduct a Welch's t-test. Levene's test, read here with deviations from the group mean, picks Student for best against good (p = 0.115) and Welch for good against least accomplished (p = 0.0073), which are the two tests the paper printed.

Move the age below 18 and every mean and test moves with it. Move it past 18 and the page refuses to test: at 19, 1 good and 3 least accomplished violinists had been interviewed at 18, and the workbook holds no estimate for years they had not yet lived. Ask for the professionals and the page refuses too. The replication interviewed 4 and ran no test on them; its supplement says: Due to the small size, we do not conduct any inferential statistics with this group. This page does not ship their rows. The 1993 study interviewed 10 and tested them once, against its best students at 18, as a further test of the relation between performance and practice: The average for middle-aged violinists is 7,336 hr, which is so close to the average of 7,410 hr for the best young violinists, that the difference is not statistically significant. (p. 380). That test needs the 1993 rows, which were never released, so this page cannot rerun it.

IV / The 1993 test, on the 2019 rows

One step comes back at nearly full strength; the other comes back reversed

The 1993 paper describes its analysis in one sentence: the hypothesized differences between the three groups of violinists are represented by two orthogonal contrasts (p. 374). That is one error term shared by two comparisons: best against good, and the average of the best and good (the paper calls them the soloist students) against the teachers. The replication describes the same analysis as two ANOVAs; the page uses the paper's own description.

Run exactly that procedure on the 39 released histories. Best and good against least accomplished comes back F(1, 36) = 11.45, p = 0.0017, against the printed 11.86. Best against good comes back F = 1.12 (p = 0.30) with the sign reversed: the good group ahead, by 1,620 hours. The first half of the claim, that students on a soloist track had practised far more than students heading for teaching, came back at nearly the printed strength, with the soloist-track students ahead by 4,476 hours. The second half, the best over the good, did not.

V / The control on the control

Could this replication have said yes?

A replication that finds nothing can only speak against what it could have found. So before reading the replication's null result as an answer, the page asks what it would have reported if the 1993 gap had been there. The replication did publish a power statement:

Our sample size is large enough to detect an effect size of η² = 0.48, which is the effect size found by Ericsson et al. [1], with greater than 99.9% power.Macnamara and Maitra (2019), p. 5.

That is the sensitivity of the three-group test, and the replication passed that test (Kruskal-Wallis chi-squared 13.90, p = 0.001). The step it reports as not replicated is best against good, and for that step it printed no sensitivity at all. So the page measures one, directly: it plants the 1993 gap into a copy of the replication's own weekly histories and runs the replication's own, unmodified analysis on the copy.

Grade A, by injection. The claimed effect goes into the control's own released data, and the control's own analysis runs again on the copy. The check trusts only that the workbook records what the violinists said. It does not lean on any sensitivity figure the replication printed, because the one it printed belongs to a different test.

At the default reading the answer is: probably not. Planted at the printed 2,109 hours, the gap comes back from the replication's own test at Student t = 1.20, p = 0.240, and only 2,489 of 10,000 resampled replications confirm it (power 0.249). The reason is spread: the 2019 best and good groups have standard deviations of 3,355 and 5,348 hours, against the 2,201 implied by the 1993 tests, so the printed gap is only d = 0.472 in the 2019 spread. A normal-theory check agrees: 0.21 at d = 0.472 with 13 a group. Power is a chance, not a yes or no, so the page's words follow a stated rule: The words follow bootstrap power: below one half the replication was unlikely to confirm the planted gap, from one half to 80% it could have, and from 80% it probably would have.

Read the claimed size without its units and the answer changes, barely. As the printed ratio of means the gap is 3,916 hours in the 2019 good group's terms, and the planted test gives p = 0.035 (power 0.595); as the printed standardized gap, d = 0.958 in the 2019 three-group spread of 3,895 hours, it is 3,732 hours, p = 0.043 (power 0.559; the normal-theory figure for this planted gap, d = 0.836 in the best-and-good spread the replication's test uses, is 0.53). By that rule the replication could have confirmed the claim at either reading, and at both it falls short of the 80% chance a replication is designed for. All three readings are live above, and the default is the one least favourable to the replication. Planting the whole 1993 pattern changes nothing about the first step (p = 0.240) and costs the second: good against least accomplished falls to p = 0.263, while the three-group test still passes (p = 0.025). That plant also drives 1 total and 65 weekly estimates below zero, which is why it is arithmetic and not a possible history.

And the test that was unlikely to say yes did say something. Test the replication's own estimate, −1,620 hours, against a true gap of 2,109: t = −2.13, p = 0.044. Once the 1993 study's own uncertainty is included the same comparison gives p = 0.063. So the replication was unlikely to confirm the printed gap, and its own estimate sits well below it. Both are true, and the page says both.

VI / The claimants' method, run on nothing

Does the 1993 procedure manufacture the claim?

The replication raised the possibility that the 1993 result was a false positive produced by the analysis:

Ericsson et al.'s [1] method of conducting two ANOVAs per question, and in particular comparing the best and good violinists but using the full sample degrees of freedom, increases the chances of finding p < 0.05.Macnamara and Maitra (2019), p. 17.

That can be tested. Feed the 1993 procedure data in which the three groups have the same true mean, and count how often it reports the claim anyway. What such a test can show turns on something the 1993 paper never printed: how spread out each of its groups was. A test that pools one error term across three groups is correctly sized when the groups share a spread, so null data built with one shared spread cannot show the worry at all. The first two families below are built that way; the last two keep each 2019 group's own spread, and the page starts on the first of those.

With one shared spread the procedure is correctly sized for best over good: it calls that step significant in 2.49% of null datasets with shuffled labels and 2.25% with the 1993 design, where a correctly sized test pointing one way should do so about 2.5% of the time. Keep each 2019 group's own spread and it is not: 5.83% with resampled residuals and 4.66% with normal draws, 1.9 to 2.3 times the nominal rate. The mechanism is the one the replication named. In 2019 the least accomplished group was the tightest (standard deviation 2,379 hours, against 3,355 and 5,348), so pooling it in shrinks the error term that judges best against good, and the step is then judged on the full sample's degrees of freedom.

Complete correspondence stays rare under every family (at most 0.11%), because the second contrast turns conservative when the tight group is the one it sets apart: alone it fires in 0.57% and 1.18% of the datasets that keep the 2019 spreads. The 1993 paper printed only the pooled spread, so which of these describes its own test cannot be settled from the record. All of this is this page's computation, on these two contrasts only, not the paper's other analyses.

VII / The second layer

What a 13-violinist replication could and could not see

1. The asymmetric instrument

Plant every gap from zero to 6,000 hours into the 2019 rows, resample, and count how often the replication's analysis confirms it (green). On the same axis, the dashed blue line is the p-value for the replication's actual estimate tested against each planted gap: where it falls below 0.05, the replication's data say the gap is not that large.

The curve is computed in your browser after the page loads.

The curve's left end is its calibration. With no gap planted it reads 0.035 (4,000 resampled replications at each of 25 gaps), against the 2.5% a correctly sized one-directional test would give: drawing 13 violinists with replacement from 13 makes the bootstrap a little generous, so the bootstrap powers on this page, if anything, overstate what the replication could see.

At the printed 2,109-hour gap the 2019 replication had about a one-in-four chance of confirming it, yet its estimate lies 2.1 standard errors below it.

The instrument is lopsided in a way neither paper says. It was unlikely to confirm a gap of the printed size, yet its own estimate rejects a gap that large (p = 0.044), though not once the 1993 study's uncertainty is included (p = 0.063). Every gap up to 1,994 hours lies inside the replication's own 95% interval for its estimate, so its data rule none of them out, and at that largest size it confirms the gap in only 23% of resampled replications. Between zero and 1,994 hours, the gaps the replication's own data cannot rule out, this replication can say almost nothing.

To give an 80% chance of confirming the gap, normal theory (two-sided .05, the page's computation) says a replication would need 72 violinists a group at the printed hours (d = 0.472 in the 2019 best-and-good spread) and 24 at the standardized reading (d = 0.836 in that spread); if the gap were d = 0.958 in the new study's own spread, as it was in 1993, 19 would do. Set the instrument to 72 a group at the printed gap and resampling gives 0.83, a little above the target, as the calibration above would predict.

2. Two studies, one gap

In hours the two studies differ at z = 1.86 (p = 0.063); in standardized units, z = 2.14 (p = 0.032). Pooled with fixed weights, the gap is 1,213 hours with a standard error of 858 (p = 0.157), or d = 0.18 with a standard error of 0.30 (p = 0.55). In standardized units each study uses the spread it has: 1993's d is the gap over the pooled spread of all three groups, the only spread it printed; 2019's is the gap over the pooled spread of its best and good groups. The pooling is this page's own, and it carries a caveat the arithmetic cannot see: these are different populations. The 1993 violinists were Berlin students with a mean age of 23.1 at interview; the 2019 violinists were Cleveland students recruited from 2014 to 2018, who reported more practice overall and far more competition entries.

3. The claimants' reply, operated

On the day the replication was published, The Guardian (21 August 2019) reported that Ericsson said the new paper actually replicated most of their findings. He said there were no objective differences between Macnamara’s best and good violinists, so no surprise they put in the same amount of practice. His co-author Ralf Krampe, the paper reported, said nothing in Macnamara’s paper made him question the original findings. He also told it: Do I believe that practice is everything and that the number of hours alone determine the level reached? No, I don’t, and But I still consider deliberate practice to be by far the most important factor. Ericsson then replied in print (Journal of Sports Sciences, 2020): the replication, he argued, did not reproduce the 1993 difference in objective performance between its best and good groups, and did not report or discuss that failure. The replication did print the comparison (successful competition entries, best against good, t = 1.55, p = 0.134, p. 11) and argued that its groups kept the same relative difference in skill (below); it did not call the comparison a failure to replicate. This page could not open the full text of that reply (the publisher returned HTTP 403), so it paraphrases it from the sentences indexed by Semantic Scholar's citation record, rather than quote it.

The 1993 paper supplies the yardstick itself: The best indicator of violin performance, besides the evaluation of the music professors, is success at open competitions. (p. 374). So put the two studies' competition records side by side.

In 1993 the best violinists had 2.9 successful entries on average against 0.6 for the good, F(1, 27) = 19.35: by the paper's own contrast arithmetic a gap of d = 1.97. In 2019 the best had 13.31 against 8.69, t = 1.55, p = 0.134, d = 0.63. The 1993 best and good groups were far further apart on the skill marker, which is Ericsson's point. And the 2019 best group still led on competitions while trailing on practice, which is the replication's; its authors wrote: While our method of recruitment followed Ericsson et al. [1] and our violinists appeared to have the same relative difference in skill from each other, they may have an overall higher level of expertise than Ericsson et al.'s [1] violinists (e.g. the current violinists had entered many more competitions than those in 1993). The page does not fit a model that turns one gap into the other: two numbers, two readings, both printed. Macnamara's own summary: Practice makes you better than you were yesterday, most of the time, she told The Guardian. But it might not make you better than your neighbour. Or the other kid in your violin class.

4. The method on nothing

Section VI above. Complete correspondence stays rare under every null the page ran (at most 0.11%), so the 1993 procedure does not make the whole claim out of nothing. Its best-over-good step alone is another matter: correctly sized when the groups share one spread (2.49%), firing 1.9 to 2.3 times as often as a correctly sized test when they keep the 2019 spreads (5.83% and 4.66%). Which of these applied in 1993, the record cannot say.

VIII / The verdict, dated

Sources

1. K. Anders Ericsson, Ralf Th. Krampe and Clemens Tesch-Römer (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review 100(3): 363 to 406. doi:10.1037/0033-295X.100.3.363
2. Brooke N. Macnamara and Megha Maitra (2019). The role of deliberate practice in expert performance: revisiting Ericsson, Krampe & Tesch-Römer (1993). Royal Society Open Science 6(8): 190327. doi:10.1098/rsos.190327. CC BY 4.0.
3. Brooke N. Macnamara and Megha Maitra (2019). Additional Methods and Results from The role of deliberate practice in expert performance: revisiting Ericsson, Krampe & Tesch-Römer (1993). The Royal Society, electronic supplementary material to Royal Society Open Science 6: 190327. doi:10.6084/m9.figshare.9250445.v1. CC BY 4.0.
4. Macnamara and Maitra, Replication and Extension: Seminal DP Study, OSF project 4595q, file kqxgj (the workbook); the same dataset deposited at Dryad, doi:10.5061/dryad.68db279, under CC0 1.0.
5. Macnamara and Maitra, preregistration, OSF registration khjs7, registered 19 April 2017.
6. K. Anders Ericsson (2020). Towards a science of the acquisition of expert performance in sports: Clarifying the differences between deliberate practice and other types of practice. Journal of Sports Sciences 38(2): 159 to 176 (online 12 November 2019). doi:10.1080/02640414.2019.1688618
7. K. Anders Ericsson and Kyle W. Harwell (2019). Deliberate practice and proposed limits on the effects of practice on the acquisition of expert performance: why the original definition matters and recommendations for future research. Frontiers in Psychology 10: 2396. doi:10.3389/fpsyg.2019.02396. CC BY 4.0.
8. K. Anders Ericsson (2014). Why expert performance is special and cannot be extrapolated from studies of performance in the general population: A response to criticisms. Intelligence 45: 81 to 103. doi:10.1016/j.intell.2013.12.001 (quoted from the in-press version)
9. Ian Sample (2019). Blow to 10,000-hour rule as study finds practice doesn’t always make perfect. The Guardian, 21 August 2019.
10. Florida State University News, 24 June 2020, notice of the death of K. Anders Ericsson (archived copy of 25 June 2020).