Post-Hoc Probability Fallacy

The post-hoc probability fallacy is the error of specifying a probability test after the fact — computing the improbability of an outcome that has already been observed, as if it had been predicted in advance — and concluding that the outcome is too remarkable to have occurred naturally. The mathematician David H. Bailey, writing on Math Scholar, calls it “a form of confirmation bias reasoning” and stresses that it is not a marginal flaw but usually fatal: “in most cases, the post-hoc fallacy completely nullifies the argument. The correct reckoning is ‘What is the probability of X occurring, given that X has been observed to occur,’ which of course is unity.” ¹

It belongs to the same family as the dwindling-probabilities problem and the measure problem documented at fine-tuning-argument, but is logically distinct from both: it concerns when and how the probability question is posed, not how the resulting terms are combined. Its natural home is any probability argument built on an already-known outcome — design arguments from biology, cosmology (fine-tuning-argument), or consciousness (psychophysical-harmony), and probability-laden apologetics more generally (bayesian-apologetics). ¹

Structure of the fallacy

1. Observe an outcome that is, in fact, already known (a rock’s measurements, a dealt hand, an existing protein sequence, the universe’s constants).
2. Enumerate a space of theoretical possibilities for that outcome and assume equiprobability across it.
3. Take the reciprocal of the space’s size as “the probability” of the outcome, declare the number vanishingly small, and conclude that the natural process under challenge cannot have produced it.

Every step after the first is defective: the possibility space is stipulated rather than empirically established; the equiprobability assumption is almost never justified (some amino acid sequences are relatively likely to emerge, while vast numbers of other sequences are not biologically possible at all); and, decisively, the test was constructed after seeing the data, so no advance prediction was ever on the table. ¹

Key point — nullification, not weakening. Because any outcome can be made to look improbable by declaring a fine-grained enough possibility space, an improbability figure by itself carries no evidential weight: “the laws of probability, when correctly applied to a post-hoc phenomenon, can say nothing one way or the other about the likelihood of the event”. This is stronger than the usual statistical complaint — the argument is not merely weakened but “completely” nullified. ¹

The illustrations

• The rock (“space aliens made this rock”). Twenty measurements of an ordinary rock yield 80 digits at four significant figures, so “the probability” of a rock matching them exactly is ~1 in 10^80 — so remote that scrutinising every planet of every star in the Milky Way, repeated across 100 billion galaxies, would still be expected to find no repeat. The conclusion “aliens made it” collapses on inspection: real rock measurements are constrained by physics and geology to lie close to neighbouring points, so the equiprobable 80-digit model is invalid; and, post hoc, some rock had to be found. ^{1 2}

• The dealt hand. A specific, nondescript 13-card hand has “approximately one in 4 x 10^21” probability, yet “there is nothing particularly remarkable about this hand of cards at all”. Declaring that “the dealer must have cheated” misreads the calculation: some hand had to be dealt, and the post-hoc computation is “completely misleading”. ^{1 2}

• Biology — the original target. Creationist and intelligent-design claims about the human alpha-globin sequence (“one in 20^141, or one in approximately 10^183”) are “dead-ringers” for the fallacy: computed after the fact on a long-known sequence; some sequence had to occur; the enumerated space is not equiprobable in reality; natural selection is directionally non-random; and wide functional tolerance means many molecules could perform the same role. ¹

• Beyond biology — finance, cosmology, Fermi’s paradox. A 2023 companion essay generalises the fallacy: finance is “deeply afflicted” through backtest overfitting (“statistical mirages”), and in cosmology and astrobiology “we only have one real data point” — one universe, one rise of technological life — which hobbles any rigorous probability reckoning. ²

Bailey’s neighbouring lesson: genuinely improbable “coincidences” happen routinely — in a class of 30, some shared birthday is 70.6% likely — so intuitions about small probabilities need calibration. ¹

Where the fallacy is guarded against

Fields most exposed to post-hoc reasoning have adopted institutional safeguards — evidence that it is treated as a first-order methodological hazard, not a technicality: ¹

• Preregistration — analysis and tests declared before data collection (clinical trials, social sciences).
• Data blinding — pseudorandom blinds subtracted only after data collection and analysis are complete (particle physics, astronomy, cosmology).
• Data holdout — models developed on part of a series, then evaluated once on held-out data; reuse of the holdout re-introduces the fallacy. The finance illustration Bailey cites: on five years of daily data, trying more than ~45 strategy variations leaves an overfit Sharpe ratio of 1.0 or higher expected by chance alone. ^{1 2}

Relevance to the fine-tuning argument

The fine-tuning case is where Bailey extends the fallacy directly — the 2020 essay was about evolutionary biology; the 2023 companion essay turns to cosmology. It inventories the standard cases (force strengths, nuclear masses, CMB anisotropy, the cosmological constant, the Higgs hierarchy problem, spatial flatness, entropy) and then concludes: “we have no evidence of these other universes, and as yet we have no conceivable means of rigorously calculating the ‘probability’ of any possible universe outcome, including ours (this is known as the ‘measure problem’ of cosmology). As with Fermi’s paradox, by definition we have only one universe to observe and analyze, and thus, by the post-hoc probability fallacy, simple-minded attempts to reckon ‘probabilities’ are doomed to failure.” ²

Read carefully, the claim is not that fine-tuning is illusory: it is that post-hoc improbability reckonings cannot settle the question. There is only one sample; any obtained parameter values can be made to appear astronomical by declaring a fine enough possibility space and assuming equiprobability over it; and the multiverse option inherits the same measure problem rather than dissolving it. On fine-tuning-argument this aligns with §1 (the measure problem) and §7 (post-hoc probability reckoning); the distinction from the “post hoc” objection already there (Carroll’s evidence-selection complaint) is worth keeping — Bailey’s fallacy concerns the after-the-fact reckoning, not the after-the-fact rhetoric. ^{2 3}

Relevance to the psychophysical harmony argument

The psychophysical-harmony argument asks why the actual psychophysical mapping is harmonious rather than chaotic: most conceivable mappings would be disharmonious, and atheism (the claim runs) gives no reason to expect a harmonious one. The post-hoc structure appears twice: (i) the space of “conceivable psychophysical laws” is enumerated by the theorist after the fact, with equiprobability assumed rather than established; (ii) there is a single sample — one actual pairing of mind and world, with no ensemble of alternative psychophysical regimes to calibrate against — so the bare improbability of the actual mapping cannot itself discriminate between hypotheses (“some mapping had to obtain”). Critics already press the hypothesis-space worry through the reference-class and combinatorial-explosion objections (see that page §2); the post-hoc fallacy is the general form of that complaint. It bites hardest against specific-mapping formulations; where the argument compares class-level likelihoods, what is needed is a well-defined measure over the alternatives plus an independent reason to expect a designer to select a harmonious one. ⁴

Relation to Bayesian reasoning

• Correct conditioning adds nothing. “The probability of X given that X has been observed” is unity; an already-known datum cannot itself update credences about how it arose. The “old evidence” problem in Bayesian confirmation theory is the formal shadow of this point (see fine-tuning-argument §1). ³
• Selection destroys the warrant. A likelihood computed for data that was used to define the test carries no evidential weight; Bailey’s institutional triple (preregistration, blinding, holdout) is the applied-statistics remedy. ¹
• It is not only a priors problem. Where subjectivity-of-priors faults the inputs to a valid computation, the post-hoc fallacy faults the computation’s setup — it can corrupt a probability argument even when priors and likelihoods would be agreed on. The two failure modes travel together in apologetic probability arguments, which is why the wiki treats them as complementary critiques. ¹

See Also

• fine-tuning-argument — cosmological design argument; §7 applies this fallacy directly
• psychophysical-harmony — consciousness-based design argument; §2(d) applies it to the harmony likelihoods
• bayesian-apologetics — apologetic probability arguments; the fallacy appears as critique 5
• subjectivity-of-priors — the co-travelling input-side critique
• dwindling-probabilities — chained-hypothesis critique; a sibling failure mode
• reasoning-methods — where Bayesian confirmation sits among the modes of inference

References

• Bailey, D. H. (2020). “Do probability arguments refute evolution?” Math Scholar (essay; page updated 30 September 2025). ¹
• Bailey, D. H. (2023). “Aliens made this rock: The post-hoc probability fallacy in biology, finance and cosmology.” Math Scholar (essay; page updated 26 July 2025). ²
• Rosenhouse, J. (2022). The Failures of Mathematical Anti-Evolutionism. Cambridge University Press. (Cited in the first essay as the book-length treatment of anti-evolution probability arguments.)
• Friederich, S. (2017; rev. 2026). “Fine-Tuning.” Stanford Encyclopedia of Philosophy. ³
• Cutter, B. & Crummett, D. (forthcoming). “Psychophysical Harmony: A New Argument for Theism.” Oxford Studies in Philosophy of Religion. ⁴

Footnotes

1. raw/articles/mathscholar-bailey-post-hoc-probability-2020.md ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³

2. raw/articles/mathscholar-bailey-aliens-made-this-rock-2023.md ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

3. raw/articles/sep-fine-tuning-friederich-2026.md ↩ ↩² ↩³

4. raw/articles/cutter-crummett-psychophysical-harmony.md ↩ ↩²