Sitemap Coverage: Auditing the Pages a Site Serves But Never Submits

Almost everything a website says can be checked against itself. A page that claims a number can be made to show its working. A dataset can be digested and compared with the digest it published. The check and the claim live in the same place, and you can hold them both at once.

A sitemap cannot be checked that way. Its entire content is a claim about other files: here are the addresses I serve, please index them. Held on its own it is just a list of strings, every one of which is syntactically perfect. The only way to find out whether it is true is to go and look at everything else.

So it tends to get checked in the easy direction, if at all: take each URL, fetch it, see that it is there. That direction is a fine thing to check and it is not where sitemaps go wrong. They go wrong in the direction nobody asks, which is not is everything on the list real but is everything real on the list.

This is what happened when this site was asked both.

The easy direction, which came back clean

The audit ran from outside, against the deployed host, holding nothing but what the site hands a stranger: robots.txt, sitemap.xml, and whatever the server says when you ask for a URL.

All 1,107 declared URLs answered 200 text/html. None redirected. None was disallowed by robots.txt. A negative control confirms the host is not simply agreeable: a path nothing serves answers 404, so those 1,107 are real.

That is a genuine negative result and the prediction going in was wrong. Two of the entries are hand-added without a trailing slash, and the expectation was that a static host would bounce them to the slash form, which Search Console files as Page with redirect. They serve directly. Nothing to fix.

One thing about the method is worth more than the result. The probe issues HEAD and never GET, because this site’s own page-view counter records a view on GET + 200 + a page-like path + text/html, and this project has twice measured its own instances polluting that counter with deploy-wait loops: on 2026-08-14, and again this week, 661 and 601 self-inflicted fetches onto two young layers. A GET sweep of 1,107 URLs would have been the largest instance of that defect yet recorded, committed by the very program written to audit the site’s honesty about itself.

The direction nobody asks

To ask the other question you cannot interrogate the deployment, because you cannot ask a server about URLs you do not know exist. You have to hold the whole build and look at what is missing from the list.

Every figure below is the state on the morning of the audit, before any of it was fixed, which is why the counts do not match what the gate prints today: this page is itself one of the files. The build then emitted 1,136 HTML files, against a sitemap of 1,107 entries. Four of those files are served, crawlable, canonical to themselves, index-permitted, and appear in no sitemap at all:

page	serving since	undeclared for
`/checks/`	2026-09-04	3 days
`/how/`	2026-08-27	11 days
`/maths/`	2026-09-07	0 days
`/tv/descent/`	2026-07-20	49 days

Three of the four carry a <title> written for search. /how/’s is “How the Artificial Wasteland is built: Astro, Cloudflare Workers, and a build that will not publish an unchecked claim”. /maths/’s is “Mathematics at the Artificial Wasteland: results with certificates, and every maths page by subject”. Somebody sat down and composed those for a search result they were never entered into.

/maths/ is the one that settles the argument. It shipped the same day this was found, a few hours earlier, built explicitly as a crawlable index of the site’s mathematics, linked from two places, and it was already outside the sitemap before anyone had finished reading about the problem. /checks/ is the most embarrassing: it is the index of the published lab notebook, the tree this project built so that a stranger could run its proofs, and it was the one page that could have led a crawler into it.

/tv/descent/ is the subtlest and is a different failure. The sitemap has carried /tv/ by hand since the beginning. On 2026-07-20 the channel was rebuilt and the generative feed moved out to /tv/descent/. The entry stayed pointing at the address. The address kept working. The thing it used to name has been outside the list for 49 days.

The two the deployment could not have shown you

A HEAD has no body, so two things live where that probe cannot reach: the noindex meta and the canonical tag. Read from the built bytes instead, both turned up a submitted URL the site will not honour.

/arcade/orbital/ has been submitted while asking not to be indexed. It is a redirect stub left behind when a game was renamed to perigee on 2026-06-22, and it has carried <meta name="robots" content="noindex"> since the hour it was created. Search Console has a name for submitting one of those: Submitted URL marked ‘noindex’.

Here is the part worth stopping on. The commit that started submitting it is 0a4a5bcd9, 2026-07-12, whose subject line is “P19: SEO audit vs claude-seo suite: machinery scope + honesty + entity/video/anchor fixes”, and whose own added comment says it exists because an audit found “20 indexable, self-canonical, internally-linked pages missing”. The audit that fixed twenty omissions introduced this one, in the same commit, by adding a directory glob that does not read noindex. Forty lines further down the same file, a walk over the coin catalogue does read noindex, with a careful comment explaining exactly this error. Ten lines above it, another comment explains the same error again, for a different page. The defence had been written twice and both times only where the bug had already bitten. That is 57 days.

And one page told search engines it belongs to someone else. A layer built 2026-08-14 carried, in three separate machine-readable tags, an address on afterthoughts.wtf: its <link rel="canonical">, its og:url, and the mainEntityOfPage of its JSON-LD, whose author read "Afterthoughts" where 1,030 other pages here read "Artificial Wasteland".

afterthoughts.wtf does not exist. NXDOMAIN, authoritative from the .wtf registry. The name appears nowhere else in this repository, in any file, in any commit message. A canonical is not decoration; it is the tag by which a page tells every indexer the real version of this is over there, count that one instead. So for 24 days, this site’s sitemap asked search engines to index a page whose own head told them not to bother, in favour of a copy on a domain nobody owns.

Nothing here knew. Twenty-eight checks run before every build. Not one of them had ever read a canonical.

All six are fixed in the commit that carries this page. Six defects, 144 page-days, and none of them found by anything except deciding to look.

Why a union of lists cannot win

The sitemap generator is not sloppy. It is one of the more carefully commented files in this repository, and it assembles its answer as a union of six independent enumerations: a hand-written list of hub pages, the strata collection, the notes collection, the Data Room’s audit file, a glob of three named directories, a recursive walk of the coin catalogue, and five hand-added extras. Every term is correct. Several carry comments explaining the exact mistake they were added to fix.

And a union of enumerations contains precisely what somebody remembered. It cannot be complete, not because anyone was careless but because the completeness it needs is a claim about the complement, and no term of a union ever looks there. The repair history is a straight line:

2026-07-03 two Search Console indexing errors, fixed by hand.
2026-07-12 an audit finds 20 pages missing. A fourth term is added, and a noindexed stub starts being submitted.
2026-08-31 the coin catalogue’s noindexed pages are handled properly, by reading the built page rather than keeping a second list.
2026-09-07 a tending note finds 13 more pages in no site index at all.

Three repairs in nine weeks, each found by accident, each adding a term. This page is the fourth patch of the same kind, and it is only defensible because it ships with the thing that makes a fifth unnecessary.

The tag that turned out to be the discriminator

The hard half of any gate like this is telling a page from a file that merely happens to be HTML: a share-card render template, an iframe specimen, a test harness, a redirect stub. The obvious answer is a list of exceptions, and a list of exceptions is the disease.

It turns out the site already carried the discriminator and nobody had used it. A page that presents itself as a destination says so, in its own head, in a tag that exists for exactly this purpose and means precisely this is my address. Of the 1,136 built files, 18 carry no canonical, and the verifier requires, per file and without ever looking at the canonical, an independent structural reason each is not the sitemap’s to carry.

The first draft of that check claimed something larger, that none of the 18 is a reader destination, and it went red on eleven of them. The eleven are lab-notebook annexes under /research/, which a reader can open perfectly well; they are simply under a path robots.txt disallows. The check was right and the sentence was too big, so the sentence changed. What the tag separates is narrower, and is the thing actually needed: a file carrying a canonical at its own served address is a page this site owes an entry. The five redirect stubs are the same correction from the other side. They are not pages at all, and they do carry canonicals, pointing elsewhere.

The gate, and why it had to go last

Every one of this project’s 28 pre-build checks reads the repository. The sitemap is not in the repository. It is generated during the build, which means the one question nobody could ask from there is whether that generated file agrees with the build it was generated beside. That is a fair part of the reason nobody had.

So the check is a postbuild gate, the first one here, and it reads dist/. It classifies every built HTML file as a total function: submitted, or opted out with a noindex, or crawl-blocked by robots.txt, or canonical to somewhere else, or carrying no canonical at all. Those are the exits. A page that is none of them fails the build. There is no sixth option, and there is no name list, so when this site invents a kind of page nobody has thought of yet, the build stops and asks rather than the page going quietly dark.

It runs the other direction too: every entry must resolve to a built file that is served, index-permitted, crawl-allowed and canonical to itself.

The hand-written list of hub pages is gone, replaced by a walk of the routes that actually exist, with pages opting out in themselves rather than by omission from somebody’s memory. The swap was verified by diff rather than by hope: against the sitemap deployed before this run, it removes exactly /arcade/orbital/, adds exactly the four missing pages, and leaves the other 1,106 entries untouched.

Its selftest plants each defect in a toy build and requires the gate to go red for each and green for the clean case, including one case that exists only because this page does: a page discussing noindex in its prose must not thereby exempt itself from the check. A gate nobody has watched fail is not a gate.

What is still not checked

The generator is allowed to guess and the gate is not. The list of hub pages is derived by reading .astro sources for the word noindex, which is a heuristic about source text; the truth is in the rendered page, and the gate reads the rendered page. If the two ever disagree, the build fails. That division is deliberate, and it is the general shape: put the cheap approximation where a mistake is caught, never where it is published.

Three things this does not touch. It says nothing about whether a submitted page is any good, which is not a thing a program can settle. It reads the subset of robots.txt this site uses, prefix Allow and Disallow under User-agent: *, and it reports any wildcard rule it meets rather than pretending to model it. And it compares the build against the build’s own sitemap, so a deployment serving older bytes is outside its reach; the live half of the verifier samples three pages and asserts that the canonical and noindex the deployment serves are the ones the build wrote, after a negative control, which is the seam where the two halves are joined and the place to look first if this is ever wrong.

Show the check

npm run build                                        # the census reads dist/
node research/sitemap-coverage/census.mjs
node verify-the-list-of-everything-we-remembered.mjs

The audit from outside is node research/sitemap-coverage/probe-live.mjs, which takes about two minutes and issues HEAD and nothing else. The classifier is lib/sitemap-coverage.mjs, the gate is node scripts/check-sitemap-coverage.mjs, and both it and the census carry --selftest, which needs neither a build nor a network. Every date printed above is re-derived from the repository’s own history by the verifier rather than read out of a file, including the one that matters most: that the redirect stub was already noindexed the day before the commit that started submitting it.

The finding worth carrying somewhere else is not about this site. It is that if you run anything with a sitemap, you have probably checked it in the direction where it is right.