Replication Crisis: The Syllabus
Replication Crisis: The Syllabus
Your back-to-school reading list for the history of replication calamities
Around this time last year (i.e. in Fall 2022), I taught a class about the replication crisis. This is a topic pretty close to my heart, because the replication crisis kicked off right around the time I started getting serious about psychology, so a big part of my education involved watching this apocalypse unfold in real time. I tried to re-create that experience for my students, just slightly sped up. We read a lot of the things I encountered at the time, and we read them in about the same order.
The class was taught as a seminar, with a simple format: for each day, students would read a couple primary sources. Then we would discuss those sources in class, where I’d give them context for the readings and help them make sense of how each reading fits in with the larger conversation. I can’t give you all that context here, but I can at least share the reading list. Here it is.
One of the most interesting parts of the replication crisis is how much of it happened on the internet, over "informal" channels, sometimes at lightning speed. The result is that the best pieces of writing on any subject are often on someone's blog. I tried to emphasize this aspect, because I think it is important. So while the syllabus does include some research papers, you’ll see that many of the assigned readings are pieces of journalism, blog posts, or even posts on social media.
Next time I teach this course, I’ll have to add readings about topics like the recent allegations surrounding Francesca Gino’s research, Gino suing Harvard and the Data Colada team in response to these allegations, more recent work on the sociology and incentives of academia, and more recent critiques of peer review. But that’s between me and the class of 2027.
Wednesday, 7 September
Overview
Readings Due:
Andrew Gelman – What has happened down here is the winds have changed
The Nib – Repeat After Me (comic)
Richard Feynman – Cargo Cult Science
Optional Readings:
Andrew Gelman – Why is the scientific replication crisis centered on psychology?
Monday, 12 September
Fabrication
Readings Due:
The Nation – Disgrace: On Marc Hauser
Chapters 1 and 6 of Faking Science by Diederik Stapel, translated by Nick Brown.
Optional Readings:
Science – Harvard Misconduct Investigation of Psychologist Released
Any other Stapel Chapters
Wednesday, 14 September
Sociology and Incentives of Academic Science
Readings Due:
Chapter 5 of Faking Science by Diederik Stapel, translated by Nick Brown. This chapter is pretty long and the first half isn't super relevant to today's topic, feel free to skim up to page 82, "I was pleased with my shovel and bucket in Chicago. Down in the basement..."
Erik Hoel – Publish and Perish
Étienne F.D. – Book Review: Making Nature
Optional Readings:
Tal Yarkoni – No, it’s not The Incentives—it’s you
Chapter 4 of Faking Science by Diederik Stapel, translated by Nick Brown. In this chapter, Stapel talks a little more about the "under the table" elements (his words) of pre-replication-crisis psychology research. Informative if you want to read more about how things used to be.
Monday, 19 September
p-Hacking and QRPs
Readings Due:
Simmons, Nelson, & Simonsohn – False-Positive Psychology
538 – Science Isn’t Broken
Sacha Epskamp – Questionable Practices by Researchers and Teenage Wizards
Optional Readings:
Data Colada – Help! Someone Thinks I p-hacked
Wednesday, 21 September
p-Hacking responses
Readings Due:
The Atlantic – The Myth of Self-Correcting Science
Pashler & Harris – Is the Replicability Crisis Overblown? Three Arguments Examined
Optional Readings:
Gelman & Loken – The garden of forking paths
Nature – Psychologists do some soul-searching
Monday, 26 September
Power Posing
Readings Due:
Watch: The Original TED Talk
Data Colada – Power Posing: Reassessing The Evidence Behind The Most Popular TED Talk
Dana Carney (the first author on the original power posing paper) – My position on “Power Poses”
Optional Readings:
The Original Power Posing Study
Wednesday, 28 September
Power Posing Discussion & Scientific Criticism
Readings Due:
The New York Times – When the Revolution Came for Amy Cuddy
Simine Vazire – Criticizing a Scientist’s Work Isn’t Bullying. It’s Science.
Data Colada – Menschplaining: Three Ideas for Civil Criticism
Optional Readings:
A new analysis from Cuddy et al. in 2017
Monday, 3 October
Statistical Power and p-Curve
Readings Due:
Data Colada – Samples Can't Be Too Large
Data Colada – MTurk vs. The Lab: Either Way We Need Big Samples
Simonsohn, Nelson, & Simmons – p-Curve Paper
Optional Readings:
Data Colada – We cannot afford to study effect size in the lab
Data Colada – "The" Effect Size Does Not Exist
Nature – Power failure: why small sample size undermines the reliability of neuroscience
Wednesday, 5 October
YOM KIPPUR NO CLASS
Monday, 10 October
OCTOBER BREAK NO CLASS
Wednesday, 12 October
Social Priming
Readings Due:
John Bargh – Nothing in Their Heads
Ed Yong – A failed replication draws a scathing personal attack from a psychology professor
Sanjay Srivastava – Some reflections on the Bargh-Doyen elderly walking priming brouhaha
Optional Readings:
Bargh, Chen, & Burrows – Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action
Doyen, Klein, Pichon, & Cleeremans – Behavioral Priming: It's All in the Mind, but Whose Mind?
Daniel Kahneman – A proposal to deal with questions about priming effects
Monday, 17 October
Feeling the Future
Readings Due:
The New York Times – Journal’s Article on ESP Is Expected to Prompt Outrage
Ritchie, Wiseman, & French – Failing the Future: Three Unsuccessful Attempts to Replicate Bem's ‘Retroactive Facilitation of Recall’ Effect
Andrew Gelman – A new Bem theory
Optional Readings:
Data Colada – Pilot-Dropping Backfires (So Daryl Bem Probably Did Not Do It)
Wednesday, 19 October
Preregistration
Readings Due:
Data Colada – Preregistration: Not just for the Empiro-zealots
Data Colada – How To Properly Preregister A Study
Optional Readings:
Data Colada – AsPredicted: Pre-registration Made Easy
Monday, 24 October
Many Labs I
Readings Due:
Klein et al. – Investigating variation in replicability: A “many labs” replication project.
Data Colada – "Many Labs" Overestimated The Importance of Hidden Moderators
Optional Readings:
Slate – Why Psychologists’ Food Fight Matters
Data from Investigating Variation in Replicability: A “Many Labs” Replication Project
Wednesday, 26 October
On Failed Replications
Readings Due:
Jason Mitchell – On the evidentiary emptiness of failed replications
Tom Stafford – What Jason Mitchell's 'On the emptiness of failed replications' gets right
Monday, 31 October
Growth Mindset
Readings Due:
BuzzFeedNews – A Mindset "Revolution" Sweeping Britain's Classrooms May Be Based On Shaky Science
Scott Alexander – Should Buzzfeed Publish Claims Which Are Explosive If True But Not Yet Proven?
Alex Tabarrok – Growth Mindset Replicates!
Optional Readings:
What did they say about it on twitter?
Wednesday, 2 November
Reproducibility Project: Psychology
Readings Due:
Open Science Collaboration – Estimating the reproducibility of psychological science
Gilbert et al. – Comment on “Estimating the reproducibility of psychological science”
Monday, 7 November
Reproducibility Responses
Readings Due:
Anderson et al. – Response to Comment on “Estimating the reproducibility of psychological science”
Gilbert et al. – A RESPONSE TO THE REPLY TO OUR TECHNICAL COMMENT ON “ESTIMATING THE REPRODUCIBILITY OF PSYCHOLOGICAL SCIENCE"
Data Colada – Evaluating Replications: 40% Full ≠ 60% Empty
Optional Readings:
Sanjay Srivastava – Evaluating a new critique of the Reproducibility Project
Gilbert et al. – More on “Estimating the Reproducibility of Psychological Science”
Wednesday, 9 November
Facial Feedback Hypothesis
Readings Due:
Bayesian Spectacles – Musings on Preregistration: The Case of the Facial Feedback Effect
Registered Replication Report: Strack, Martin, & Stepper (1988)
Monday, 14 November
Ego Depletion
Readings Due:
Scott Alexander – Book Review: Willpower
Carter & McCullough – Publication bias and the limited strength model of self-control: has the evidence for ego depletion been overestimated?
Hagger et al. – A Multilab Preregistered Replication of the Ego-Depletion Effect
Optional Readings:
Simine Vazire – fifty million frenchmen can eat it
Carter, Kofler, Forster, & McCullough – A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource
Lurquin et al. – No Evidence of the Ego-Depletion Effect across Task Characteristics and Individual Differences: A Pre-Registered Study
Wednesday, 16 November
Many Labs II
Readings Due:
Klein et al. – Many Labs 2: Investigating Variation in Replicability Across Samples and Settings
After 10 Years, ‘Many Labs’ Comes to an End – But Its Success Is Replicable
Monday, 21 November
Brian Wansink
Readings Due:
Science – Cornell nutrition scientist resigns after retractions and research misconduct finding
Brian Wansink – The Grad Student Who Never Said "No" (Take a look at the comments on this blog post as well)
WIRED – This Thanksgiving, Ditch the Food Psychology
Wednesday, 23 November
THANKSGIVING NO CLASS
Monday, 28 November
Data Thugs
Readings Due:
Science – Meet the ‘data thugs' out to expose shoddy and questionable research
Nature – Meet this super-spotter of duplicated images in science papers
Optional Readings:
James Heathers – I Quit. And I’m OK With That
Joe Hilgard – Smell you later
Wednesday, 30 November
Dan Ariely
Readings Due:
Data Colada – Evidence of Fraud in an Influential Field Experiment About Dishonesty
BuzzFeedNews – Dan Ariely Retracts Honesty Study Based On Fake Data
Optional Readings:
Science – Fraudulent data raise questions about superstar honesty researcher
Forbes – An Influential Study Of Dishonesty Was Dishonest
Monday, 5 December
Generalizability and Paradigms
Readings Due:
Tal Yarkoni – The Generalizability Crisis
Thomas Kuhn – What Are Scientific Revolutions?
Wednesday, 7 December
Peer Review & Publication
Readings Due:
Michael Nielsen – Three myths about scientific peer review
Vox – Meet Alexandra Elbakyan, the researcher who's breaking the law to make science free for all
Simine Vazire – Opening the Black Box of Peer Review
Optional Readings:
Vox – Let's stop pretending peer review works
WIRED – Peer-Reviewed Scientific Journals Don't Really Do Their Job
Smith (former editor of the British Medical Journal) – Classical peer review: an empty gun
This looks great. I haven't read the large majority of these - a reminder that I don't know nearly as much about the replication crisis as I like to pretend I do. (James Heathers is awesome, by the way). Your autobiographical approach (having the syllabus track your own developing critical consciousness about the problem) is intriguing; I have long been interested in ways to integrate personal elements more deliberately into research or academic contexts without getting too loosey-goosey about it.
Chris Chamber's "Seven Deadly Sins of Psychology" is also a nice overview (we read that in my stats class). I'm also wondering if Simine Vazire might be worth foregrounding even a little more,; as I recall her "Sometimes I'm Wrong" blog was a key site for those early discussions and traces her own emergence as a major leader of the Open Science movement.
Nothing from Retraction Watch? Your students should get familiar with it. Bjoern Brembs on peer review and glamourmagz.