P-Values: Are we using a flawed statistical tool?

P-values show up in almost every scientific paper, yet they’re one of the most misunderstood ideas in statistics. In this episode, we break from our usual journal-club format to unpack what a p-value really is, why researchers have fought about it for a century, and how that famous 0.05 cutoff became enshrined in science. Along the way, we share stories from our own papers—from a Nature feature that helped reshape the debate to a statistical sleuthing project that uncovered a faulty method in sports science. The result: a behind-the-scenes look at how one statistical tool has shaped the culture of science itself.

Statistical topics
• Bayesian statistics
• Confidence intervals
• Effect size vs. statistical significance
• Fisher’s conception of p-values
• Frequentist perspective
• Magnitude-Based Inference (MBI)
• Multiple testing / multiple comparisons
• Neyman-Pearson hypothesis testing framework
• P-hacking
• Posterior probabilities
• Preregistration and registered reports
• Prior probabilities
• P-values
• Researcher degrees of freedom
• Significance thresholds (p < 0.05)
• Simulation-based inference
• Statistical power
• Statistical significance
• Transparency in research
• Type I error (false positive)
• Type II error (false negative)
• Winner’s Curse

Methodological morals
• “If p-values tell us the probability the null is true, then octopuses are psychic.”
• “Statistical tools don't fool us, blind faith in them does.”

References
• Nuzzo R. Scientific method: statistical errors. (pubmed.ncbi.nlm.nih.gov/24522584) Nature. 2014 Feb 13;506(7487):150-2. doi: 10.1038/506150a.
• Nuzzo, R., 2015. Scientists perturbed by loss of stat tools to sift research fudge from fact (scientificamerican.com/article/scientists-pert…). Scientific American, pp.16-18.
• Nuzzo RL. The inverse fallacy and interpreting P values (pubmed.ncbi.nlm.nih.gov/25724848). PM&R. 2015 Mar;7(3):311-4. doi: 10.1016/j.pmrj.2015.02.011. Epub 2015 Feb 25.
• Nuzzo, R., 2015. Probability wars (www-sciencedirect-com.stanford.idm.oclc.org/sc…). New Scientist, 225(3012), pp.38-41.
• Sainani KL. Putting P values in perspective (pubmed.ncbi.nlm.nih.gov/19769922). PM&R. 2009 Sep;1(9):873-7. doi: 10.1016/j.pmrj.2009.07.003.
• Sainani KL. Clinical versus statistical significance (pubmed.ncbi.nlm.nih.gov/22732155). PM&R. 2012 Jun;4(6):442-5. doi: 10.1016/j.pmrj.2012.04.014.
• McLaughlin MJ, Sainani KL. Bonferroni, Holm, and Hochberg corrections: fun names, serious changes to p values. (pubmed.ncbi.nlm.nih.gov/24769263) PM&R. 2014 Jun;6(6):544-6. doi: 10.1016/j.pmrj.2014.04.006. Epub 2014 Apr 22.
• Sainani KL. The Problem with "Magnitude-based Inference". (pubmed.ncbi.nlm.nih.gov/29683920) Med Sci Sports Exerc. 2018 Oct;50(10):2166-2176. doi: 10.1249/MSS.0000000000001645.
• Sainani KL, Lohse KR, Jones PR, Vickers A. Magnitude-based Inference is not Bayesian and is not a valid method of inference. (pmc.ncbi.nlm.nih.gov/articles/PMC6684445) Scand J Med Sci Sports. 2019 Sep;29(9):1428-1436. doi: 10.1111/sms.13491.
• Lohse KR, Sainani KL, Taylor JA, Butson ML, Knight EJ, Vickers AJ. Systematic review of the use of "magnitude-based inference" in sports science and medicine. (pubmed.ncbi.nlm.nih.gov/32589653) PLoS One. 2020 Jun 26;15(6):e0235318. doi: 10.1371/journal.pone.0235318.
• Wasserstein, R.L. and Lazar, N.A., 2016. The ASA statement on p-values: context, process, and purpose. (tandfonline.com/doi/full/10.1080/00031305.2016…) The American Statistician, 70(2), pp.129-133.

Kristin and Regina’s online courses:

Demystifying Data: A Modern Approach to Statistical Understanding (online.stanford.edu/courses/som-xche0033-demys…)
Clinical Trials: Design, Strategy, and Analysis (online.stanford.edu/courses/som-xche0030-clini…)
Medical Statistics Certificate Program (online.stanford.edu/programs/medical-statistic…)
Writing in the Sciences (coursera.org/learn/sciwrite)
Epidemiology and Clinical Research Graduate Certificate Program (online.stanford.edu/programs/epidemiology-and-…)
Programs that we teach in:
Epidemiology and Clinical Research Graduate Certificate Program (online.stanford.edu/programs/epidemiology-and-…)

Find us on:
Kristin - LinkedIn (linkedin.com/in/kristin-sainani-642b5914) &Twitter/X (x.com/KristinSainani)
Regina -LinkedIn (linkedin.com/in/reginanuzzo) &reginanuzzo.com/ReginaNuzzo.com (reginanuzzo.com)

• () - Intro & claim of the episode

• () - Why p-values matter in science

• () - What is a p-value? (ESP guessing game)

• () - Big vs. small p-values (psychic octopus example)

• () - Significance thresholds and the 0.05 rule

• () - Regina’s Nature paper on p-values

• () - Misconceptions about p-values

• () - Fisher vs. Neyman-Pearson (history & feud)

• () - Botox analogy and type I vs. type II errors

• () - Dating app analogies for false positives/negatives

• () - How the 0.05 cutoff got enshrined

• () - Misinterpretations: statistical vs. practical significance

• () - Effect size, sample size, and “statistically discernible”

• () - P-hacking and researcher degrees of freedom

• () - Transparency, preregistration, and open science

• () - The 0.05 cutoff trap (p = 0.049 vs 0.051)

• () - The biggest misinterpretation: what p-values actually mean

• () - Paul the psychic octopus (worked example)

• () - Why Bayesian statistics differ

• () - Why aren’t we all Bayesian? (probability wars)

• () - The ASA p-value statement (behind the scenes)

• () - Key principles from the ASA white paper

• () - Wrapping up Regina’s paper

• () - Kristin’s paper on sports science (MBI)

• () - What MBI is and how it spread

• () - How Kristin got pulled in (Christie Aschwanden & FiveThirtyEight)

• () - Critiques of MBI and “Bayesian monster” rebuttal

• () - Spreadsheet autopsies (Welsh & Knight)

• () - Cherry juice example (why MBI misleads)

• () - Rebuttals and smoke & mirrors from MBI advocates

• () - Winner’s Curse and small samples

• () - Twitter fights & “establishment statistician”

• () - Cult-like following & Matrix red pill analogy

• () - Wrap-up

P-Values: Are we using a flawed statistical tool?

Kapitelklicken springt direkt hin

Shownotes