Deciding under incomplete evidence
You run a batch of tests on an evolving idea. They pass — but you know the tests don't exercise every possibility, so "it passed" isn't certainty, it's a statement of confidence about the cases you didn't run. From that you make one call: advance the idea to the next step, or stay and redo this step until it's ready. The honest answer depends on which statistic you compute from the partial results — and every statistic has situations where it quietly lies. This page shows two — a percentile over a measured quantity and a Wilson score interval over a pass rate — how they work, where they fail, and the alternatives.
Every chart is computed live in your browser from seeded samples and exact binomial math — nothing here is hand-drawn.
Some results aren't pass/fail — they're a number per case (an error size, a deviation, a quality score), and the cases you sampled form a distribution. You can't judge the idea on "the number," so you judge a percentile: p95 is the value 95% of cases came in under; p99 is the worst 1% you saw. Why a percentile and not the mean is the whole game — the mean hides the tail, and the tail (plus the cases you never sampled) is what bites later. The concrete example here is a per-case error — how far off each answer is, as a percent (0–100%, a hard floor and ceiling).
Pick a distribution. Same ~mean each time — watch p95 and especially p99 move.
Same true distribution, resampled 400 times at each window size. Each dot is one window's estimate; the spread is the sampling noise you'd gate on.
p50 barely moves with n; p99's spread only tightens slowly — trust a small-sample p99 and you stay-and-redo on a phantom tail. (Hence: demand a minimum number of cases before you believe a percentile at all.)
The pass/fail results ask a binomial question: of n test cases,
k passed — is the true pass rate, over all the cases you didn't run, at
least the bar you set, say 0.99? The obvious answer is the point estimate
p̂ = k/n, compared straight to the line. It is simple and it is fragile: near 0.99 with a
small sample, p̂ is noisy, so the same true idea lands above the line (advance) or below it
(redo the step) on luck alone.
An idea whose true pass rate is exactly the bar (0.990). 300 independent test
batches of n = 150; each dot is one batch's k/n. The decision line is 0.99.
Instead of a bare point, compute a confidence interval for the true pass rate and
decide on the interval: advance only if the lower bound
clears the bar, redo the step only if the upper bound is below it,
otherwise gather more cases — the evidence isn't conclusive yet. Wilson is the standard
choice over the textbook Wald interval in exactly this regime (p near 1, modest
n): it never leaves [0, 1] and stays calibrated.
z²/n terms pull the center off p̂ toward ½ and keep
the whole interval inside [0, 1] — so it stays honest exactly where Wald falls apart.Three intervals as the sample grows. The dashed line is the bar. Note where each lower bound finally clears it — and that Wald's upper bound runs past 1.0 (impossible).
Wilson lower bound on a perfect sample (k = n) is n/(n+z²). Where it
crosses the bar is the minimum clean cases before you can advance.
Coverage = how often the interval actually contains the true rate; a genuine 95% interval should cover 95% of the time. Here it's computed exactly as the true rate approaches 1. Wald undercovers badly near the boundary; Wilson hugs the target.
Wald dips far under the 95% line right where our bar lives — its "95%" interval is often really 80–90%. Wilson stays close.
2000 simulated decision points under the same logic. What each rule does with the same idea.
At the bar the naive yes/no sends you back to redo the step 24% of the time and advances the rest on thin evidence; the Wilson rule holds 90%, refusing to over-claim. On a genuinely weak idea both send you back — Wilson is robust, not lax.
Everything so far judged one batch of tests in isolation. But you re-test as the idea evolves, so results arrive as a time series, and the hardest case is a flaky failure: the idea is genuinely fine, but every so often a test misfires and the measured rate briefly craters, then clears on its own. How you weigh results over time decides whether you ride it out or restart the step for nothing.
The naive instinct is to restart the step the moment results dip below the bar. Watch what that does. The run below has three short flaky dips (the idea is fine, a test just misfired) and then, near the end, one real regression that stays down. The honest fix isn't a fancier interval — it's requiring the dip to be sustained across repeated tests before you act.
xₜ gets weight
α (here ~0.3); everything older is multiplied by (1 − α) every step, so the
past decays smoothly instead of being remembered at full weight and then dropped. Small α = long
memory (steady, slow); large α = short memory (reactive, jumpy).D results, so it's ignored; a real
regression stays below the bar past D, so it's caught. D is the knob that
trades reaction speed against false restarts.One time-decayed (EWMA) signal over repeated tests. A restart-on-first-dip rule acts on every blip; a restart-on-sustained-dip rule waits D consecutive results below the bar, so it ignores flaky dips but still catches the regression that doesn't clear.
Wilson and the percentile aren't the only options — they're the ones that fit a per-batch pass rate and a measured-quantity tail. Here's the neighborhood, and when each earns its place.
k passed out of n. The Bayesian posterior is a Beta(k+1, n−k+1) density; its 95% credible interval and the Wilson interval nearly coincide here — different philosophy, similar bound.
| Method | What it estimates | Strength | Fails when… |
|---|---|---|---|
| Point estimate k/n | the raw pass rate | trivial, unbiased | small n near 0/1 — flips on sampling noise (needless restart & premature advance) |
| Wald interval | CI, normal approximation | simple closed form | near 0/1 it undercovers and can leave [0,1] — the wrong tool at a 0.99 bar |
| Wilson score | CI, score method | calibrated near 0/1, stays in [0,1], cheap | assumes independent cases; correlated failures shrink the effective n (optimistic) |
| Clopper–Pearson | exact CI (binomial) | guaranteed ≥ nominal coverage | conservative — wider than needed, so it's slow to let you advance |
| Bayesian Beta-Binomial | full posterior on the rate | priors, credible intervals, natural "not sure yet" | answer depends on the prior; needs justifying to a reviewer |
| Percentile p95/p99 | a measured-quantity tail | captures the tail the mean hides | needs ≥~100 cases for p99; unstable on small samples |
| EWMA / time-decay | a recency-weighted signal | forgets cleared dips; handles flaky transients | a tuning knob (the decay) that trades sensitivity for lag |
| Two-sample (vs baseline) | idea − baseline difference | adapts to each problem's own noise floor | needs a trustworthy known-good baseline to compare against |
| CUSUM / SPRT | sequential change-point | fastest detection of a real shift | more parameters; tuned for change-detection, not a fixed pass bar |
Everything on this page is a small corner of uncertainty quantification (UQ): the discipline of putting an honest number on how much you can trust a conclusion drawn from limited data. The distinction that runs underneath it is aleatory vs epistemic uncertainty. The noise in a pass rate from sampling is aleatory — irreducible for a fixed sample, but it shrinks as you run more cases, and that's exactly what a Wilson or Clopper–Pearson interval quantifies. The "the tests don't cover every possibility" caveat is epistemic — a gap in what you measured at all — and no amount of the same sampling closes it; that needs a different lever (comparing to a known-good baseline, broadening test coverage, or distribution-free methods like conformal prediction).
So a confidence interval, a Bayesian credible interval, a coverage curve, and a conformal prediction set are all answering one question in different dialects: given what I could measure, how sure am I? The canonical references below are good next stops.
The same statistics power any advance-or-restart decision made on incomplete tests — a canary rollout, an A/B test, a model-quality gate, an experiment. Every chart here is computed client-side with a seeded PRNG and exact binomial coverage — nothing hand-drawn; view source, it's one file.