How to Read a Study

Teach me to evaluate a scientific study myself, and use a real one I bring, not a hypothetical. Start with the concepts, then apply them to my paper.

The study I want to examine: [PASTE A LINK, AN ABSTRACT, OR THE HEADLINE AND CLAIM YOU SAW]

PART ONE: THE TOOLKIT. Teach these one at a time, checking I have each before continuing, and make me state each definition back in my own words:
- What a p-value actually means, and the four things people think it means that it does not. Nearly everyone, including many working scientists in surveys on this, gets this wrong, so confront my version directly rather than assuming I have it.
- Effect size versus statistical significance, and why a real effect can be too small to matter while a large effect can fail to reach significance.
- Statistical power, and why underpowered studies produce both false negatives and, less intuitively, exaggerated positives when they do find something.
- Confidence intervals, and why they carry more information than a yes-or-no result.
- The garden of forking paths: how honest researchers making reasonable choices about exclusions, measures, and subgroups can produce a false positive without any intent to cheat.
- Preregistration, and what it does and does not fix.

PART TWO: THE CRISIS THAT TAUGHT US THIS. Anchor it in real numbers instead of a general mood. When a large collaboration attempted to reproduce a hundred psychology studies, around a third replicated and effect sizes came in near half the originals; a similar attempt in preclinical cancer biology could not even attempt three quarters of its planned replications because the published methods were too thin to follow. On publication bias, the sharpest case is antidepressant trials, where the published record looked overwhelmingly positive while the regulator's file, which included the unpublished trials, was closer to an even split. Cover which fields fare better and why, and be specific about what has improved since, including preregistration and the finding that journals using registered reports publish far fewer positive results than conventional ones, because the story is not one of unrelieved decline.

PART THREE: THE CHECKLIST, APPLIED. Now go through my study and answer each of these explicitly:
1. What exactly was measured, and is it the thing the headline claims? Watch for a proxy standing in for the interesting outcome.
2. How many participants or observations, and was that enough to detect the claimed effect?
3. Is this a randomized experiment or an observational study, and does the language of the claim exceed what the design supports?
4. What is the effect size in terms I would notice in real life, not in standard deviations?
5. Who funded it, and does the funder have an interest in the direction of the result?
6. Is this a single study or a replicated finding, and what did the previous work show?
7. What comparison is missing? Look for the control that would be inconvenient.
8. Does the abstract's claim match what the results section actually reports?

PART FOUR: THE VERDICT. Give me a rating: how much should this change my beliefs, from not at all to substantially, with reasoning. Distinguish between "this is wrong", "this is probably true but overstated", and "this is fine and the coverage of it is wrong", because they are different failures with different fixes.

Rules: if you cannot open the full paper, state that up front and limit your claims to the abstract. On the funding and prior-literature items especially, tell me whether you are reading the paper or reasoning from memory. Do not manufacture criticisms to seem rigorous, and say plainly when a study is simply good.