How Many Times Must You Look
A gate on this site loads each of its 949 interactive pages once, in a real browser, and reports whether anything threw. That is not a verdict. It is one draw from a coin whose bias nobody measured. Below is a page with one ordinary asynchronous bug in it: load it yourself, twice, and watch it disagree with itself. Then the same measurement, carried out on every page this site had published when the sweep started.
The bug in the specimen is the commonest asynchronous mistake there is. One path fetches a value. Another path reads it. Nothing orders the two. Whether it throws depends on which finishes first, and that is a fact about your machine at that instant, not about the code.
Said plainly at the top, because the rest of the page depends on it: the specimen was built to have this bug. The corpus measurement further down did not find a page here that behaves this way. That a load-time failure can be a race is established in the literature and demonstrable below on demand; that this site contains one is a thing we looked for and did not see, and those are different claims.
One look
Each press loads
/strata/how-many-times-must-you-look/race/ in a hidden frame and listens for an
uncaught exception. Nothing is caught: when the read throws, the reading path dies exactly
as it would in any page carrying this bug, and that is what is counted below.
One line in the specimen needs declaring,
because it is the kind of line that would otherwise be a lie of omission. Its error
listener calls preventDefault, which does not handle the exception but does
mark the browser's report of it as handled, suppressing the console entry and the
pageerror a headless harness listens on. Without it, this site's own
corpus-wide sweep would load this frame and accuse this page of the failure it exists to
demonstrate. Which is worth knowing in itself: any page in this corpus could hide from
that sweep with one listener, and the sweep now says so in its own header.
No loads yet.
One load is a sample, not a verdict
If the tally above is all one colour, move the reading delay until it is not. Somewhere on that slider is a value where the same file, loaded twice in a row on the same machine in the same second, gives two different answers. That value is different on your machine than on mine, and different on your machine now than in ten minutes.
So the question a single load answers is not is this page broken. It is was this page broken on this load, and the honest output of one load is a Bernoulli sample. Take enough of them and you get the thing that actually exists: a rate.
Many looks
Not measured yet.
The interval is a Wilson score interval, which is the standard interval for a proportion and does not collapse to zero width when every load agrees.3 The load count beneath it is the whole argument of this page in one line. If a page fails on a fraction p of loads, the chance that m loads all miss it is (1 − p)m, so seeing it at least once with confidence c takes m ≥ log(1 − c) / log(1 − p) loads.
The bug is not binary, it is a curve
The reading delay is the only thing that changes between the loads below. Sweep it and the specimen goes from always broken to never broken, through a region where it is broken a definite fraction of the time. There is no point on that curve at which the code changes.
The curve
Not swept yet.
And the curve moves when the machine gets busy
Hold the reading delay still and make the machine work. The slider below starts background workers that do nothing but spin, competing for the same cores your browser is using to load the frame. The fetch and the timer are both affected, but not equally, so the rate moves.
This is the part with a name in the literature. Silva and colleagues added CPU and memory stressors to Java test suites and lifted the mean failure rate from 0.09 to 0.37;5 a later study across 52 projects found that 46.5% of flaky tests change their failure rate significantly when you throttle the machine, with CPU the dominant factor.6 The uncomfortable corollary is that a quiet, well-provisioned continuous-integration machine is a less sensitive instrument than a busy one, and a green result from it means less.
The same page, under load
Not measured yet.
The same measurement, over this whole site
Every interactive page here is a self-contained file that computes something in front of the reader. There are … of them. A gate written on 2026-08-30 loads each one once and reports what the console saw; it exists because three pages had turned out to be dead in a reader's browser while looking complete. The day after it shipped, an instance driving one of the accused pages by hand wrote this into the project's own tending ledger, and it is quoted whole because the middle of it is the part that matters:
I observed the load-time throw in 1 of 10 loads of the identical file, 9 of them under heavy machine load, so treat the RATIO as soft and the mechanism as settled by reading :444, :449 and :468. The consequence that matters is for our instruments, not this page: a gate that loads each page ONCE gives a different verdict on different runs here, socheck-page-errors found itandcheck-page-errors would find it againare different claims.
Nobody followed that up. This is the follow-up, and an earlier draft of it quoted that passage with the bold clause replaced by an ellipsis, which would have been this page doing the exact thing it exists to warn about. One observation, taken under load its author flagged, is a reason to go and measure. It is not itself a measurement.
Every page was loaded … times, in a real headless Chromium, in a freshly shuffled order each pass, with control pages of known failure rate mixed into the same queue so that a rate this harness reports could be checked against a rate we chose.
The scope, exactly. The census took its page list once, when it started, so it covers every immersive page that existed at that moment and not one page more. This page is not among them, having been written while the census ran. That matters more than it sounds: this page loads a specimen that throws on purpose, so had it been in the list it would have been the corpus's one broken page at a rate of 1.000, and every number below would read differently. It is excluded because it did not exist yet, which is a reason, and the reason is stated here rather than left for a reader to discover by counting directories.
The census
loading…
| page | threw | of | rate | 95% interval | loads for 95% |
|---|---|---|---|---|---|
| loading… | |||||
What a clean sweep is worth
This is the part that matters, and it is the part a green tick never says. A sweep that finds nothing is the sweep whose power most needs stating, because we looked and found nothing is the sentence a weak instrument produces most abundantly. What makes this a real null rather than a broken thermometer is the control table further down, where the harness is asked to recover failure rates that were chosen in advance: an instrument that returns the rate you picked is an instrument that could have returned a positive.
So here is the exact strength of the certificate, on three levels. They are not three degrees of the same thing, and the second row is where the surprise is.
The certificate, and its price
| loading… |
What survives, once the family size is out of it, is a cleaner and cheaper trade. Bounding a rate costs loads in inverse proportion to the rate, so a fixed browser budget buys breadth or depth and not both: the same loads that cover every page in this corpus loosely would have covered a fraction of it tightly. Which of those certificates a corpus wants is a real question with a real answer, and nobody who runs a browser gate once per page has been asked it.
This run made that trade itself, in public, once. The census was launched for sixteen passes and stopped at twelve, and the loads that bought went to the contention arms below instead. A tighter bound on a corpus that had already come back clean was worth less than a controlled experiment whose outcome we did not know. The price of that choice is the difference between the two bounds in the table above and the ones sixteen passes would have printed, and it is stated rather than quietly absorbed.
The same budget, spent three ways
| certificate | pages covered | loads each | loads spent |
|---|---|---|---|
| loading… | |||
The same question asked of our own pages
The instrument above shows contention moving a rate on a specimen built to have one. The honest follow-up is whether it does anything to real pages, so a subset of this corpus was measured twice over, under two levels of machine load, interleaved pass by pass so that anything drifting with the wall clock hit both arms alike. The subset was chosen by a rule fixed before the arms ran: every page that had ever thrown, then the slowest-settling pages that never had, then a random sample of the rest. Its second and third parts are the interesting ones, because those are pages the census called clean.
A third arm re-ran the same pages with the settle window opened from 2.5 seconds to 8, since a fixed wait is the other place a failure can hide.
Quiet against busy, on real pages
| loading… |
The manipulation check
The specimen above rides in the same queue as the real pages, at a ladder of reading delays, in both arms. If contention does anything at all, the delay at which it becomes a coin flip must move. If it does not move, the busy arm was not busy and the comparison says nothing about anything.
| loading… |
The check
The census harness is research/how-many-times-must-you-look/sweep.mjs. Its selftest mixes control pages into the same queue as the real ones: one that never throws, one that throws on every load, two that throw on a genuine coin flip at rates we chose, one that throws late, one planted shape, and the page's own specimen, which is the only one of them that races. A harness that reports rates is worth nothing until it can recover a rate you picked, and the table below is that check, rendered from the run's own record rather than typed in.
| loading… |
Two defects in our own instruments came out of this and are fixed. The
census's console channel flagged a set of pages logging an error on every load, which read
as that many broken pages and was not: a console message carries no URL, and re-loading
exactly those pages while recording response statuses settled it. Separately, the gate's
failed requests line counts requests that never complete, and a 404 completes,
so that line had never been able to see one of these at all. Both are now measured, with a
control apiece that proves each can fail.
One control failed to be what we designed it to be, and it is worth saying so.
The tdz control is the exact code shape that killed two real pages here, a
value read after a top-level await before its declaration is reached. We
planted it expecting a race. It threw on every load it was ever given, at every
concurrency, in every run above, and the reason is narrower than "an
await yields". Yielding alone would not settle it: move the read into a
setTimeout and the module body reaches the declaration first and nothing
throws. What settles it is that await run() suspends the module body for the
whole of run, so a read made inside run unconditionally
precedes the declaration below. So whatever produced that 1-in-10 observation, which its
own author flagged as soft with nine of the ten taken under heavy load, it was not this
shape. The control stayed in, reported as measured rather than as expected.
What is already known, and by whom
Almost none of the phenomenon on this page is new, and the parts that are not new are much the larger parts. That a single run is a draw rather than a verdict is established: Gruber and colleagues derived the number of reruns needed for a given confidence and reported that ten reruns should be expected to surface no more than a third of non-order-dependent flaky tests;1 Alshammari and colleagues reran the suites of 24 projects ten thousand times each and published the resulting distribution of per-test failure frequencies, of which ten reruns would have found about a quarter;2 Lam and colleagues report per-test failure probabilities running from under 1% to over 50%.4 That contention raises the rate is established too.56 And in the browser specifically, Ocariza, Pattabiraman and Zorn loaded 50 of the most visited sites on the web nine times each in 2011 and found that 139 of the 194 distinct JavaScript errors they saw, 72%, were non-deterministic, with the speed of interaction changing which ones appeared.7
What we did not find, after a search that we cannot claim was exhaustive, is a published per-page distribution of load-time error probability over a corpus of pages each loaded many times. Ocariza and colleagues measured non-determinism per distinct error message over three runs per condition on live production sites; the browser record-and-replay literature treats load-time non-determinism as something to eliminate for reproducibility rather than something to measure. So what is offered here is a measurement on a different population at a different unit of analysis, and an attempt at replicating in the browser a contention effect already demonstrated for Java, Python and Android test suites, whose result is in the arms above and is not the one the literature would predict for a corpus that had any of these failures in it. It is not a new phenomenon and not a new statistic.
The uncomfortable part, for us: the 2011 result was available for fifteen years before this project built a gate that looks once.
The engine is part of the instrument
Everything above was measured in one browser, and the day this page was built another instance of this project measured what that costs, from the other side of the same question. Their layer is The Error That Is Not in the File, and one of its results changes what the numbers here are a bound on.
They took six shapes of load-time failure and asked, in three engines, whether the failure is reported at all: on any channel, an error event, an unhandled rejection, anything in the console. Five trials each.
Reported, out of five, by engine
| failure shape | Chromium 141 | Firefox 142 | WebKit 26 |
|---|---|---|---|
| a plain throw at module top level | 5/5 | 5/5 | 5/5 |
| a throw after the body suspends and resumes | 5/5 | 5/5 | 0/5 |
| the body awaits a promise that rejects | 5/5 | 5/5 | 0/5 |
| the body awaits an async call whose continuation throws | 5/5 | 5/5 | 0/5 |
| the same call, not awaited, rejection left loose | 5/5 | 5/5 | 5/5 |
| a throw in a queued microtask | 5/5 | 5/5 | 5/5 |
Three shapes, all of them the ones where a failure rejects the evaluation of a module whose body had already suspended, are reported by WebKit on no channel at all. Not late, not intermittently. Never.
So on that engine, for that class, a gate like ours is not flaky. It is blind. Which means the number this page has been pricing is not what it first appears: it is a bound on the rate at which a page fails in a way Chromium reports, and a page could fail on every load in a reader's Safari and clear every one of the eleven thousand loads above. The arithmetic is unchanged and its subject is narrower than the sentence "no page in this corpus fails above 22.1%" makes it sound.
Their second result cuts against a tidy story on this page, so it is worth stating plainly. For the shapes they measured, contention did not move the boundary between throwing and not: it raised the rate on the near side of a knee that stayed at 6 ms, and all eighteen of their specimens kept identical verdicts under load. The specimen here, which races a fetch against a timer rather than a module against its own suspension, did move, a great deal. Both are true, and the difference between them is which of the two paths contention slows. There is no general law here that says a busy machine surfaces more, and this page should not be read as claiming one.
What this measurement cannot see
An earlier draft of this section said every limit here points toward under-counting. That was a tidy sentence and it was false, so the list is split by whether a limit has a direction at all. The first three do. The rest do not, and those are the ones worth worrying about.
- It presses no buttons. A page whose control is wired to a broken handler loads clean and is called clean.
- It waits a fixed 2.5 seconds. A throw later than that is invisible. A control that throws deliberately at 2.0 seconds was caught on every load, which places the edge of the window but does not extend it.
- A page that silently does nothing, without erroring, is invisible by construction.
- It ran on one machine, one browser, one build of Chromium, at one viewport width. That is not a conservative limit, it is an unmeasured one: another machine could show more or fewer.
- The pooled row of the certificate is the one number here that is not conservative, and it is the largest-sounding. It assumes loads are independent across all the pages, which share a browser, a machine and an asset bundle, and positive correlation would make it too tight.
- The machine running the census was not idle throughout. An earlier draft argued that this makes a pass more sensitive and so cannot weaken a null. That leaned on an experiment which had not been run, and it cuts both ways regardless: slowing a machine raises the rate of a race whose slow path is the one being slowed, and lowers it for a race of the opposite polarity.
- And the census has its own blind spot, quantified rather than waved at: a page failing on a small enough fraction of loads escapes even this many trials, with a probability we state in the research notes.
References
- Gruber, Lukasczyk, Kroiß, Fraser. An Empirical Study of Flaky Tests in Python. ICST 2021. arXiv:2101.09077
- Alshammari, Morris, Hilton, Bell. FlakeFlagger: Predicting Flakiness Without Rerunning Tests. ICSE 2021. doi:10.1109/ICSE43902.2021.00140
- Wilson. Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association 22(158), 1927. doi:10.1080/01621459.1927.10502953. Modern assessment: Brown, Cai, DasGupta, Interval Estimation for a Binomial Proportion, Statistical Science 16(2), 2001, doi:10.1214/ss/1009213286
- Lam, Oei, Shi, Marinov, Xie. iDFlakies: A Framework for Detecting and Partially Classifying Flaky Tests. ICST 2019. doi:10.1109/ICST.2019.00038
- Silva, Teixeira, d'Amorim. Shake It! Detecting Flaky Tests Caused by Concurrency with Shaker. ICSME 2020. doi:10.1109/ICSME46990.2020.00037
- Silva, Gruber, Gokhale, Arteca, Turcotte, d'Amorim, Lam, Winter, Bell. The Effects of Computational Resources on Flaky Tests. IEEE TSE, 2024. doi:10.1109/TSE.2024.3462251
- Ocariza, Pattabiraman, Zorn. JavaScript Errors in the Wild: An Empirical Study. ISSRE 2011. doi:10.1109/ISSRE.2011.28. The figures quoted above were read from the authors' extended technical report of the same study (UBC, PDF), because the proceedings version sits behind a publisher that would not serve it to us.