What does your rule actually do?

This is not a sample-size calculator. Bring it the rule you already have. It enumerates every possible count, shows which counts reject, and sums their probabilities exactly under the null and the alternative you named.

Everything runs in this page. Nothing is uploaded. The result is exact under the model you typed, not automatically exact for the real study that produced your data.

Rule syntax and conventions

model: fisher
n1: 4
n2: 4
m: 4
theta1: 4
rule: fisher
tail: greater
alpha: 0.05

How to check this yourself

For the binomial model, make one spreadsheet row for every integer k from 0 through n. Put BINOM.DIST(k,n,p,FALSE) in the probability column. Confirm the whole column sums to one, then sum exactly the rows listed in the rejection region. Use p0 for size and p1 for power.

To verify a score region, calculate (k-n*p0)/sqrt(n*p0*(1-p0)) and apply the named tail. For score-cc, subtract 0.5 from the magnitude of k-n*p0 before dividing. For Wald, calculate (k/n-p0)/sqrt((k/n)*(1-k/n)/n); with an interior p0, counts 0 and n have zero estimated standard error and an infinite signed statistic. The z rules reject only when the signed or absolute statistic, as required by the tail, is strictly greater than the cutoff.

For fixed-margin Fisher, list x from max(0,m-n2) through min(n1,m). Give each row weight COMBIN(n1,x)*COMBIN(n2,m-x) and divide by the sum of all weights. A one-sided p-value is the matching inclusive tail. A two-sided probability-ordering p-value sums every row whose null weight is no greater than the observed row's weight. Confirm that exactly the rows with p-value at most alpha are listed. For conditional power, multiply each row by theta1^x before normalising, then sum the listed rows. This is also the procedure printed inside every certificate.

What a pass proves, and what it does not

A binomial pass proves that the complete region shown has the displayed size and power under independent Bernoulli trials at exactly the probabilities shown. A Fisher pass proves the analogous conditional statement at the fixed margins and odds ratio shown.

It does not prove independence, a stable probability, honest or complete data, no optional stopping, the alternative you chose, or that this is the best rule. Fisher's conditional result does not become unconditional by dropping the word conditional.

Why a discrete rule can have no size at all

The reason any of this matters is that a count is a whole number, and a whole number cannot be tuned. Take Fisher's tea taster: eight cups, four of each, she names four. Under no discrimination the number she gets right is hypergeometric, and the only upper tail probabilities that exist in this design are 1/70, 17/70, 53/70, 69/70 and 1. That is the complete list. There is no rule of size 0.05 here, and no arithmetic will produce one. The famous test, reject only when all four are right, has attained size 1/70, which is 0.0142857, twenty-nine per cent of the five per cent it is usually described as. Fisher spells the whole distribution out himself, in the same passage where he introduces the experiment. Load the tea-tasting example above and the region and the fraction are on screen; the five numbers above are one column of a spreadsheet away.

Everything else on this page is that observation applied to cases where the arithmetic is less obvious. A rule stated as a threshold on a continuous statistic still has to land on integers in the end, and the integer it lands on moves as the sample size moves, which is why the attained size of a normal-approximation rule saws up and down rather than settling.

Deliberate limits

• The binomial limit is 2,000 trials. The page still lists all 2,001 outcomes in its certificate. On the repository host, generation plus the first full certificate check took 1.24 seconds at 2,000, versus 0.39 seconds at 1,000 and 5.21 seconds at 3,000. Browser and device times will differ.
• Fisher input is limited to 1,000 people per group and 1,001 possible fixed-margin tables. This page checks one fixed set of margins, not all possible margins.
• Barnard and Boschloo tests are absent. Their unconditional size requires a proved supremum over a nuisance probability. A grid, however fine, would not certify that claim.
• The two-sample z rule is absent, for exactly the same reason, and this is the one worth reading twice. "Reject when the pooled two-proportion z exceeds 1.96" is the rule most people arrive here holding, and it looks like something this page could sum term by term. It is not. Under the null the two rates are equal, but their common value is unknown, so the size is the worst case over every value that rate could take, and summing at one assumed rate gives a rejection probability rather than a size. The error is not small and it does not have a reliable sign. At 20 in each arm, the rejection probability at a common rate of 0.5 is 0.0425, which reads as comfortably conservative, while the true size, taken over the whole range, is 0.0534, which is above nominal. The maximum sits near 0.31, nowhere a person would think to look. A page that printed the first number under the word "exact" would be handing a study designer a false reassurance in the one place it matters most.
• The familiar survey number 385 is absent. It is usually a confidence-interval margin-of-error calculation, not a finite rejection rule with this certificate.
• Continuous t tests are absent. Their probabilities require special-function integration and do not share this finite, term-by-term certificate.
• A rule that rejects nothing is accepted, and its green tick means less than the others. Ask for reject: k >= 21 at n = 20 and the region is empty, the size and the power are exactly zero, and the certificate holds, because zero is the true answer for a rule that never rejects. But both sums then run over no terms at all, so that particular pass has examined nothing. It is disclosed here rather than refused, because the honest answer to "what does my rule do" is sometimes "nothing", and a tool that will not say so is worse than one that does.