Ground truth · archaeology · a grave file and its printed tables

A Coin in Every Third Grave

Emma Brownlee published her early medieval grave file and its R code under an open licence, which is what makes this check possible. Read under its own headers the file puts a coin in 31.6 per cent of the graves of four regions; with those regions' flag columns moved back one place the figure is 2.4 per cent, and only then do all 11 coefficients and 11 P-values of her Table 13 come back at the printed precision, so the article's results stand and the shift is in the download. Then the question a grave-level P-value skips: shuffle end dates among cemeteries, and 13 to 15 of the 61 category trends Tables 13 to 21 print as significant stay below 0.05.

Reading the record...

In 2021 Emma Brownlee published a study of how the burial of objects with the dead declined across early medieval north-western Europe, in the open-access journal Internet Archaeology. With it she deposited the grave-by-grave file behind the study and the R code that computes its tables, under a CC BY 3.0 licence. Deposits like that are rarer than they should be, and everything on this page depends on this one.

Four regions' flag columns sit one place off their headers; realigning them restores Brownlee's Table 13, returning 11 coefficients and 11 P-values at printed precision while reducing the apparent coin share from 31.6 per cent to 2.4 per cent.

So, plainly, before anything else. The shift is in the file anyone downloads. With those four regions' columns read in their right places, Brownlee's Table 13 comes back in full, every coefficient and every P-value at the printed precision (its printed sample size does not, and the page says why below), and across the four regions' own category tables 32 of their 44 printed coefficients come back exactly, against 0 read as labelled. Her tables were computed from correctly aligned data, and the article's results stand. Nothing here says her analysis, her conclusions or her archaeology are wrong, and nothing here can say how the shift happened. What would settle it for certain is a corrected file from the author or the journal.

It still matters, because the file is the part that gets reused. Run the published R code on the published file and, for unfurnished burial in those four regions, it returns the opposite sign to the printed coefficient in 6 of 7 cases.

I

One grave, twelve yes-or-no columns

The file has one row per grave: a cemetery, a grave number, a count of objects (number_grave_goods), then twelve yes-or-no columns, then a date and a region. The first of the twelve is headed none, the unfurnished flag. The other eleven are Brownlee's categories, from dress_accessories to coins.

This is grave 16 at Fellbach-Schmiden, in northern Alamannia. Its object count is 0. Read under the file's own headers, it has no dress accessories, no jewellery, no weapons, no vessels: nothing at all, except a coin.

Flip the switch. Nothing in the file changes; the same digits are read differently. In the four regions each column holds the category named in the next header along, and the last column, headed coins, holds the unfurnished flag. Grave 16 becomes what a grave with no objects ought to be: unfurnished. Across those regions the share of graves with a coin falls from 31.6 per cent to 2.4 per cent (and 31.8 per cent of their graves have no objects at all, which is where the coins were coming from). Brownlee's unfurnished trend for northern Alamannia goes from -0.123 to 0.193, the value she printed, and Table 13 goes from 0 to 11 of its eleven coefficients.

Only four regions

The shift is not everywhere. Choose which regions to read rotated. Every region's own category table is recomputed from the file each time, so you can watch Kent or Bavaria break instead.

Read as printed, Bavaria's Table 17 comes back with 11 of its eleven coefficients and 11 of its eleven P-values; rotate Bavaria and 0 of them do. Northern Alamannia is its mirror image.

II

Three routes to the same shift

A switch that makes numbers agree could be a coincidence somebody went looking for. So here are three separate reasons to think the shift is real, and none of them leans on the others.

Route one: the file checks itself

Every grave carries its count of objects as well as the twelve flags, and the count is not part of the shifted block. A grave with no objects should have no category marked, and a grave with a category marked should have at least one object.

In the four regions, “no category marked” agrees with “no objects” in 64.3 per cent of graves as labelled and 99.2 per cent rotated. In the other seven regions it is 99.7 per cent as labelled and 55.1 per cent rotated. Read as labelled, 2,752 graves in the four regions have more categories marked than objects counted; rotated, 76 do. Of those regions' 65 cemeteries, 63 read better rotated.

Two cemeteries in the four regions are labelled on the chart. Hollogne-aux-Pierres has 0 graves with no objects among its 15, so this test cannot tell the readings apart there. Eichstetten, with 169 graves, agrees better as labelled, 71.6 per cent against 64.5. The next route shows why.

Route two: the author's own counts

The article's interactive Figure 1 loads a table of sites that the data page does not list, with each site's number of graves, of unfurnished graves and of graves with each category. Joined to the grave file by name, with accents, case and punctuation ignored, 58 sites in the four regions match. For 52 of them all eleven category counts equal the file's counts rotated; for 0, as labelled. Their count of unfurnished graves equals the column headed coins at 54 sites and the column headed none at 0. Of 105 matched sites in the other regions, 73 equal the file as labelled and 0 rotated.

Brownlee's cemetery summary file agrees. Its category percentages sit closer to the rotated reading for 57 of the 57 cemeteries it shares with the four regions, and closer to the labelled reading for 107 of 107 elsewhere. The summaries, too, were built from aligned data.

And Eichstetten. The site table's category counts equal the rotated reading in 11 of eleven categories, and its 33 unfurnished graves equal the rotated flag, 33. What disagrees is the object count: the file gives 43 Eichstetten graves with no objects and a mean of 2.882 objects per grave, where the site table gives 2.402. Eichstetten fails the zero-object test on its object column, not on its flags. Why its object counts differ between the two files is a question this page cannot answer.

Route three: Brownlee's printed tables

The third route is the anchor, and it is Brownlee's own numbers. The engine reads from her R code which cemeteries each test used, computes Kendall's tau-b and the P-value that R's cor.test() reports, and compares both with the values printed in the article. Nothing printed is fed in.

Realigned, with every other choice below at its default, Table 13 (northern Alamannia) gives back 11 of eleven coefficients and 11 of eleven P-values at the printed precision. Its printed sample size, N=2622, does not come back: the subset that reproduces those values has 2,449 graves. Table 2's unfurnished trend for the period of furnished burial comes back as 0.122 with P 1.046e-11, where Brownlee printed 0.122 and 1.046e-11. Table 14 (southern Alamannia) gives 10 coefficients and 11 P-values; as labelled, 0 coefficients. Table 17 (Bavaria), outside the four, is the control: exact as labelled, broken if rotated. Across the eleven category tables, 77 of the 121 printed coefficients come back exactly and 102 within 0.002 when realigned; as labelled, 45 and 65.

What the printed tables do not settle

• Tables 2 to 12 come back less cleanly. Of their 18 unfurnished coefficients, 3 come back exactly from the realigned flag, and 5 if an unfurnished grave is read as one with no objects. Table 4 (Bavaria) prints 0.104, which is the no-objects reading (0.104), not the flag (0.112). No single reading reproduces all of them, so the check above offers all three.
• The Spearman coefficients land close but mostly not on the printed digit (northern Alamannia computes -0.24282 against a printed -0.244), so the page never uses them as the anchor.
• Printed sample sizes differ from the R code's subsets. Table 13's heading says Change over the period of furnished burial (N=2622), and the subset that reproduces every coefficient has 2,449 graves. Table 21 prints Change over the period of furnished burial (N=2547) for Kent, where the R code's subset has 2,723.
• Table 9 prints 0.159 for the number of objects in eastern Frankia; the file gives -0.159, and the article's text calls the trend a decrease. Table 14 prints -0.006 for amulets; the file gives 0.006, with the same P-value.
• The R code leaves "Staunch Meadows" out of the Anglian furnished-period tests, but the file spells the cemetery Staunch Meadow, so as written it stays in: 1,570 graves instead of 1,394, a Spearman coefficient of -0.474 instead of -0.330 (printed -0.330), and 0 of Table 15's coefficients instead of 7.
• The R file holds 155 tests for 157 printed cells. It ends, with no final line break, after the Low Countries amulets test, so Table 23's last 2 rows have no line of code. The page computes them on the same subset as the rows above and marks them.
• In the article's HTML, Table 17's yellow highlighting is inverted: it marks the rows that are not significant and leaves the significant ones plain, so all 11 rows disagree with their own P-values. Its heading reads Change over the period of unfurnished burial (N=2712), where the other category tables say furnished. In Table 10 the furnished-period unfurnished cell repeats the overall one, and the R code computes it on the overall subset (line 142).

What the record leaves open

• After realignment, 117 graves still have an unfurnished flag that disagrees with a zero object count, 60 of them at Eichstetten. They are kept, not dropped, and listed in the check panel.
• Outside the four regions, 42 graves have a none flag that disagrees with their object count, and no rotation explains them. (6,122 of the file's 6,164 such disagreements are in the four regions.)
• The article says: The following analysis is based on a dataset of 205 cemeteries in use during the 6th to 8th centuries, from six modern-day countries. The grave file names 183 cemeteries, and has 1 grave with no cemetery name at all (grave 425, in Burgundy). The site table assigns 186 sites to the eleven regions and marks 32 “Not analysed further”, and its graves column for those regions sums to 27,168, against 26,168 rows in the grave file. The page does not know why these counts differ.
• 212 graves have no grave number. 12 graves in 3 cemeteries are filed a few years off the date every other grave in their cemetery carries; the choice “one date per cemetery” reads them at their cemetery's date.
• The journal's server sends the data files with a last-modified date of 28 June 2023. That is a fact about a web server. On its own it says nothing about when or how the columns moved.

III

How many of the trends are about cemeteries?

The shift is only in the download, and the article's tables were computed from aligned data, so nothing about early medieval burial changes. And with twenty-six thousand graves, every trend is overwhelmingly significant anyway.

The objection, in this page's words

The first half is what the three routes above show, for the values they reproduce, and none of them can say how the shift got into the file; a corrected file from the author or the journal would settle that. The second half is worth testing, and the test is a question about statistics, not about Brownlee.

Every grave in the file carries its cemetery's date, the date the cemetery went out of use. Brownlee is clear about it: all of the graves from one cemetery were labelled with the same date, which in many cases would have been later than the actual date of each grave, and Yet this is unavoidable if we want to make use of the large numbers of undated graves, which are usually the poorly furnished ones. She notes that this weakens the coefficients. It has a second effect. A grave-level P-value treats Table 13's 2,449 graves as that many independent observations, but the date in that test is carried by just 15 cemeteries, sharing 8 dates between them.

So test the same coefficient another way. Keep every grave in its own cemetery, shuffle the cemeteries' end dates among themselves, and recompute Brownlee's own statistic each time. The shuffle P-value is the share of shuffles that give a coefficient at least as strong as the observed one. It asks whether the observed coefficient is unusual among those the same cemeteries give when their dates are handed out at random; it does not count how many cemeteries carry a trend. A pattern that a few large cemeteries make by happening to be early or late is one that shuffled dates tend to make too. Brownlee's own caution points the same way: This is because the statistical tests below are reliant on a rapid turnover of cemeteries for accurate results.

Every shuffle figure written into this section's sentences comes from one default run: seed 1, 9,999 shuffles per coefficient, judged by the shuffle. The explorer and the census below start from that run and recompute for whatever you choose; these sentences stay as they are, and the census names the default run beside yours.

First, northern Alamannia's dress accessories. The grave-level P-value is 3.757e-06, as printed (3.757e-06). Shuffle the end dates 9,999 times among its 15 cemeteries and a coefficient at least as strong turns up with P = 0.354 ± 0.005 (the second number is the Monte Carlo standard error). Those fifteen cemeteries cannot tell that trend apart from shuffled dates.

Pick any printed coefficient, what to measure on its cemeteries, and the unit to judge it at. Brownlee's code tests counts with Spearman's rho and yes-or-no flags with Kendall's tau-b, and the explorer does the same. When she printed no coefficient for a measure on those cemeteries (the number of categories present never appears in her tables), the readout says the value is the page's own.

The picture shows why. The bars are the coefficients that shuffled dates produce. The curve is the spread the grave-level P-value assumes, far narrower, because it counts graves as independent. The dots are the cemeteries, which are the things that actually carry the dates.

The census

Now run the shuffle on every coefficient Brownlee printed. Of the 61 category trends that Tables 13 to 21 print with P below 0.05, 14 stay below it when the end dates are shuffled among cemeteries (seed 1, 9,999 shuffles). The count moves slightly with the seed, from 13 to 15 across ten seeds, because 2 trends sit on the line; that range was computed by the verifier, not in your browser. Across Tables 2 to 23, 29 of 97 printed significant trends hold. Judge the same 61 at the grave instead, as R reports it, and 60 are below the line; take one point per cemetery, a different statistic, and 17 are. The census below can judge its verdicts at any of the three.

One coefficient goes the other way. Amulets in Kent, printed with P 0.06037 in Table 21, fall to 0.010 ± 0.001 when shuffled among Kent's cemeteries, and Brownlee's text does list amulets among Kent's decreases: In Kent (Figure 63), many different object types saw no clear decline in their use, but dress accessories, fittings, and amulets all decreased in the frequency with which they were placed in graves. Her statistically significant increase in vessels in Kent (printed P 0.01288) shuffles to 0.184 ± 0.004 across 17 cemeteries with 6 dates. The Anglian decline in dress accessories holds easily: 0.0002 ± 0.0001.

Read a failure here carefully. Burgundy's Table 20 rests on 10 cemeteries with 7 dates, which can be handed out in only 151,200 distinct ways, so the page tries every one of them and there is no Monte Carlo error at all. 0 of its 11 printed significant trends stay below the line; its P-values run from 0.081 to 0.370. That does not mean Burgundy saw no change. It means these cemeteries cannot tell the trend apart from shuffled dates. The Lower Rhine is the case Brownlee herself flagged: When so many cemeteries were in use for such a long period of time, this region will appear to have more static practice than it did in reality. Among its 11 cemeteries, 0 of its 9 printed significant trends hold.

The census table also shows a cemetery-level coefficient, and it is a different statistic: each cemetery counts once, whatever its size. It can disagree with the shuffle test (for northern Alamannia's dress accessories it is -0.403, with shuffle P 0.044) because a small cemetery weighs as much as a large one. It sits beside the test, not in place of it.

A warning about ranks

One more thing the file invites, and should not be trusted with. Figure 12 plots 14 Kent cemeteries, 2,341 graves. Rank them by mean objects per grave and Mill Hill (3.7500, from 80 graves) edges out Gilton Ash (3.7232, from 112). Rank them by categories present and Gilton Ash leads, 2.7679 against 2.5250. Resample each cemetery's own graves 2,000 times and Mill Hill comes out above Gilton Ash in a share of 0.519 of the resamples on objects and 0.148 on categories, and the full observed order recurs in 0.0 and 0.3 per cent of them. The swap at the top is noise, and this page never ranks cemeteries as a result.

IV

What the record cannot tell you

Some questions this file invites, it cannot answer. The instrument refuses them, and says why.