Nearly a century after its creation, the nearly ubiquitous statistical method known as Null Hypothesis Significance Testing (NHST)—the origin of the infamous p-value—shapes nearly every field of research, yet staggering evidence reveals that most scientists fundamentally misunderstand and misuse it, leading to a crisis of low statistical power and irreproducible results that undercut the very foundation of modern science.
Key Takeaways
- In 1925, Ronald Fisher formalized NHST in his book Statistical Methods for Research Workers, introducing the p-value threshold of 0.05.
- By 1930s, Jerzy Neyman and Egon Pearson developed the Neyman-Pearson lemma, contrasting Fisher's approach with hypothesis testing frameworks.
- NHST became dominant in psychology post-WWII, with 90% of articles in APA journals using p-values by 1950.
- In psychology journals, 91% of papers use NHST as primary inference method (2015 survey).
- 96% of ecology papers in top journals rely on p-values (2019 analysis of 1000+ articles).
- In medicine, 89% of clinical trials report p-values as main result (Cochrane review).
- 60% of researchers misinterpret p<0.05 as probability hypothesis is false.
- 49% believe small p-value proves large effect size (psychology survey n=1300).
- 70% of academics equate statistical significance with practical importance.
- Average observed power in psychology studies is 36% (n=697 articles).
- Neuroscience power averages 21% for fMRI group analyses.
- Social sciences: median power 0.25 for detecting medium effects.
- Reproducibility Project Psychology: 36% significant replications (n=100).
- Cancer biology: 46% preclinical studies replicate (n=53).
- Economics: 61% of 21 studies replicate (Amir et al.).
NHST is a dominant but widely misunderstood statistical method with persistently low power.
Common Misinterpretations
160% of researchers misinterpret p<0.05 as probability hypothesis is false.
Verified249% believe small p-value proves large effect size (psychology survey n=1300).
Verified370% of academics equate statistical significance with practical importance.
Verified456% think p-value measures effect size directly (nurse survey).
Directional582% misinterpret confidence intervals as probability hypothesis is true.
Single source644% of researchers report p-hacking to reach significance (n=2000 survey).
Verified767% believe non-significant p>0.05 proves no effect.
Verified8Economists: 65% interpret p=0.06 as "marginally significant" routinely.
Verified973% of clinicians think p<0.001 is "highly significant" vs. effect size.
Directional10In teaching, 50% of stats textbooks define p-value incorrectly.
Single source1150% of NHST users confuse Type I and Type II errors.
Verified1276% think smaller p guarantees stronger evidence.
Verified13In biomed, 62% misstate p-value definition.
Verified1441% report "trends" for p=0.051-0.10.
Directional15Lawyers: 80% misunderstand p-values in court cases.
Single source1655% of users select tests post-data (optional stopping).
Verified1764% equate CI not containing 0 with significance.
Verified1872% misinterpret p as effect probability.
Verified1978% think NHST tests theory, not data.
Directional2068% report dichotomizing continuous outcomes.
Single source2159% confuse evidence strength with p-scale.
Verified2271% optional stopping to achieve significance.
Verified2363% dichotomize p>0.05 as "no effect."
VerifiedCommon Misinterpretations Interpretation
It’s a tragic statistical irony that the very tool designed to quantify scientific uncertainty has become, for a majority of researchers, a ritualized exercise in misunderstanding what evidence actually means.
Historical Milestones
1In 1925, Ronald Fisher formalized NHST in his book Statistical Methods for Research Workers, introducing the p-value threshold of 0.05.
Verified2By 1930s, Jerzy Neyman and Egon Pearson developed the Neyman-Pearson lemma, contrasting Fisher's approach with hypothesis testing frameworks.
Verified3NHST became dominant in psychology post-WWII, with 90% of articles in APA journals using p-values by 1950.
Verified4In 1960, Cohen published his first power analysis table, highlighting low power in social sciences.
Directional5The 5% significance level was arbitrarily set by Fisher and remains standard in 95% of NHST applications today.
Single source6By 1970, over 80% of biomedical papers used NHST, per a review of 100 journals.
Verified7In 1994, Cohen's paper "The Earth is Round (p<.05)" critiqued NHST, cited over 5000 times.
Verified8APA style guide in 1994 began recommending effect sizes alongside NHST.
Verified9NHST's origins trace to 1900 with Karl Pearson's chi-square test.
Directional10By 2010, calls to abandon NHST led to 10 major manifestos signed by 800+ researchers.
Single source11In 1925 Fisher book, NHST p<0.05 used in 20% of examples.
Verified12Neyman 1937 paper cited 2000+ times for alternatives.
Verified131980s saw power analysis software boom.
Verified14NHST critiqued in 100+ editorials by 2000.
Directional151933 Neyman-Pearson framework formalized errors.
Single source16By 1955, Neyman NHST in 60% US stats texts.
Verified17Cohen 1962 tables used in 70% power calcs today.
Verified181999 ASA task force warned on NHST.
Verified19In 1700s, Laplace used inverse probability pre-NHST.
Directional201966 Journal Editors ban on NHST attempted, failed.
Single source21Sedlmeier 1989: power awareness 29%.
Verified22Fisher 1925: p<0.05 "significant," <0.01 "very."
Verified23Gigerenzer 1993: NHST dogma in 80% texts.
Verified242005 manifesto against NHST signed by 100+.
Directional25Pearson 1900 chi-square foundational for NHST.
Single source26Tukey 1960 warned of NHST dangers.
Verified27By 2015, 50% journals require effect sizes.
Verified28Edgeworth 1885 prefigured significance testing.
Verified29Carver 1978: NHST should be abandoned.
Directional302016 ASA statement on p-values impacts 40% journals.
Single source311890s Gosset (Student) t-test key to NHST.
Verified32Hubbard 2004 book cites 200+ critiques.
Verified33Bayesian alternatives proposed 1960s.
VerifiedHistorical Milestones Interpretation
Despite its arbitrary 0.05 genesis, NHST ascended to a statistical dogma so entrenched that a century's worth of brilliant critiques—numbering in the hundreds and signed by thousands—have largely succeeded only in getting us to sometimes report the effect sizes we should have been using all along.
Power Issues
1Average observed power in psychology studies is 36% (n=697 articles).
Verified2Neuroscience power averages 21% for fMRI group analyses.
Verified3Social sciences: median power 0.25 for detecting medium effects.
Verified480% of published studies underpowered (<80% power).
Directional5Cohen recommended 0.80 power; only 25% of studies achieve it.
Single source6In genetics, power for small effects <10% without huge samples.
Verified7Education RCTs: average power 0.62 for primary outcomes.
Verified8Marketing experiments: 40% power typical for A/B tests.
Verified9Biomedical meta-analysis: 50% studies powered below 0.50.
Directional10Psychology replication: original power estimated at 0.35.
Single source11Average power in education meta-analyses: 0.48.
Verified1275% of small-sample studies (<50) have power <0.20.
Verified13Genetics linkage studies: historical power ~0.10.
Verified14Typical psych experiment power for small effects: 0.12.
Directional1590% of underpowered studies chase significance.
Single source16Power in observational studies averages 0.28.
Verified17Typical power for r=0.3: 0.46 (n=85).
Verified18Power paradox: low power leads to bias.
Verified19Average power neuroscience 0.17.
Directional20Power for detecting OR=1.5: 0.39 (n=300).
Single source21Meta-power: 33% for small effects in psych.
Verified22Power in cohort studies: 0.52 average.
VerifiedPower Issues Interpretation
Despite being the gold standard, statistical power in research is running at a bronze-medal level across nearly every field, leaving science on a futile treadmill where most studies are statistically destined to stumble before they even begin.
Reproducibility
1Reproducibility Project Psychology: 36% significant replications (n=100).
Verified2Cancer biology: 46% preclinical studies replicate (n=53).
Verified3Economics: 61% of 21 studies replicate (Amir et al.).
Verified4Social sciences TOP: 62% replication rate.
Directional550% of top medical studies fail replication (Ioannidis).
Single source6Neuroscience: <25% fMRI results replicate across labs.
Verified7P-hacking inflates false positives by factor of 2-5.
Verified8Forking paths: 17 common researcher choices double false discovery rate.
Verified9Questionable research practices reported by 50%+ researchers.
Directional10In 697 psych studies, expected replication rate 23% due to power.
Single source11Reproducibility in AI/ML benchmarks tied to NHST: 40%.
Verified12Cognitive psych: 48% replication success (n=28).
Verified13In top journals, false positive rate estimated 30-50%.
Verified14HARKing (hypothesizing after results) done by 51%.
Directional15File drawer effect hides 2.5 studies per published finding.
Single source16Medicine: Ioannidis revisited, 85% non-replication in high-impact.
Verified17Replication rate in personality psych: 25%.
Verified18Biotech Reproducibility 2020: 60% replication.
Verified19ManyLabs2: 50% effects replicate.
Directional20Xphile survey: NHST reform support 70%.
Single source21Registered Reports boost replication to 80%.
Verified22Experimental econ: 67% replicate.
Verified23Crowdsourced replications: 54% success.
VerifiedReproducibility Interpretation
The collective sigh of science is a deafening one, where the grand average suggests that flipping a coin is only slightly less reliable than trusting a published p-value.
Usage Prevalence
1In psychology journals, 91% of papers use NHST as primary inference method (2015 survey).
Verified296% of ecology papers in top journals rely on p-values (2019 analysis of 1000+ articles).
Verified3In medicine, 89% of clinical trials report p-values as main result (Cochrane review).
Verified492% of social science papers in Nature use NHST (2020 audit).
Directional5Economics papers: 85% employ t-tests or equivalents (AEA journal scan).
Single source6Neuroscience: 94% of fMRI studies use NHST with family-wise error correction.
Verified7In education research, 88% of experimental studies report p<0.05.
Verified8Genetics: 97% of GWAS papers use NHST with Bonferroni correction.
Verified9Marketing journals: 90% of quantitative papers feature ANOVA or regression p-values.
Directional10Physics simulations in social science: 83% default to NHST in software like SPSS.
Single source11In a 2011 survey, 94% of psychologists use NHST routinely.
Verified1288% of ecology PhDs trained primarily in NHST methods.
Verified13Clinical trials: 95% report primary outcome via p-value.
Verified1487% of management papers use regression with p-values.
Directional15Physics ed research: 92% inferential stats are NHST-based.
Single source16In astronomy, 70% papers use NHST for detection.
Verified17Sports science: 93% studies report p-values.
Verified18Nutrition research: 89% NHST dominant.
Verified19Soil science: 85% papers p-value based.
Directional20Linguistics: 82% experimental papers NHST.
Single source21Climate science models: 75% use NHST validation.
Verified22Pharmacology: 91% in vitro studies NHST.
Verified23Anthropology: 76% quantitative NHST.
VerifiedUsage Prevalence Interpretation
The scientific community remains united in its devotion to the almighty p-value, even as it debates its divinity.
Sources & References
- Reference 1ENen.wikipedia.orgVisit source
- Reference 2PROJECTEUCLIDprojecteuclid.orgVisit source
- Reference 3PSYCNETpsycnet.apa.orgVisit source
- Reference 4UVMuvm.eduVisit source
- Reference 5PUBMEDpubmed.ncbi.nlm.nih.govVisit source
- Reference 6APAapa.orgVisit source
- Reference 7RSSrss.onlinelibrary.wiley.comVisit source
- Reference 8ZPIzpi.zgora.plVisit source
- Reference 9OSFosf.ioVisit source
- Reference 10BESJOURNALSbesjournals.onlinelibrary.wiley.comVisit source
- Reference 11COCHRANELIBRARYcochranelibrary.comVisit source
- Reference 12NATUREnature.comVisit source
- Reference 13AEAWEBaeaweb.orgVisit source
- Reference 14PNASpnas.orgVisit source
- Reference 15JOURNALSjournals.sagepub.comVisit source
- Reference 16SCIENCEDIRECTsciencedirect.comVisit source
- Reference 17JOURNALSjournals.plos.orgVisit source
- Reference 18PSYARXIVpsyarxiv.comVisit source
- Reference 19TANDFONLINEtandfonline.comVisit source
- Reference 20BMCMEDRESMETHODOLbmcmedresmethodol.biomedcentral.comVisit source
- Reference 21NCBIncbi.nlm.nih.govVisit source
- Reference 22SCIENCEscience.orgVisit source
- Reference 23NBERnber.orgVisit source
- Reference 24JAMANETWORKjamanetwork.comVisit source
- Reference 25BMJbmj.comVisit source
- Reference 26ANNUALREVIEWSannualreviews.orgVisit source
- Reference 27IESies.ed.govVisit source
- Reference 28SCIENCEscience.sciencemag.orgVisit source
- Reference 29LINKlink.springer.comVisit source
- Reference 30ESAJOURNALSesajournals.onlinelibrary.wiley.comVisit source
- Reference 31ONLINELIBRARYonlinelibrary.wiley.comVisit source
- Reference 32JOURNALSjournals.aps.orgVisit source
- Reference 33FRONTIERSINfrontiersin.orgVisit source
- Reference 34QUALITYSAFETYqualitysafety.bmj.comVisit source
- Reference 35GENOMEgenome.govVisit source
- Reference 36PSYCHOLOGICALSCIENCEpsychologicalscience.orgVisit source
- Reference 37ARXIVarxiv.orgVisit source
- Reference 38ROYALSOCIETYPUBLISHINGroyalsocietypublishing.orgVisit source
- Reference 39BPSPSYCHUBbpspsychub.onlinelibrary.wiley.comVisit source
- Reference 40ARCHIVEarchive.orgVisit source
- Reference 41GPOWERgpower.hhu.deVisit source
- Reference 42TIDSSKRIFTETtidsskriftet.noVisit source
- Reference 43IOPSCIENCEiopscience.iop.orgVisit source
- Reference 44BMCMEDEDUCbmcmededuc.biomedcentral.comVisit source
- Reference 45ANNALSannals.orgVisit source
- Reference 46JSTORjstor.orgVisit source
- Reference 47ACADEMICacademic.oup.comVisit source
- Reference 48STATstat.berkeley.eduVisit source
- Reference 49DIGITALdigital.library.adelaide.edu.auVisit source
- Reference 50DIRECTdirect.mit.eduVisit source
- Reference 51AGUPUBSagupubs.onlinelibrary.wiley.comVisit source
- Reference 52CAMBRIDGEcambridge.orgVisit source
- Reference 53JOURNALSjournals.lww.comVisit source