SlideShare a Scribd company logo
1 of 44
Download to read offline
Covering Medical Studies:
How Not to Get It Wrong
            AHCJ
      Boston, March 2013


         Ivan Oransky, MD
  Executive Editor, Reuters Health
   Co-Founder, Retraction Watch
           @ivanoransky
Covering Medical Studies: How Not to Get It Wrong
Put Down That Coffee!
Put Down That Coffee!
STUDY LINKS COFFEE USE TO PANCREAS CANCER
New York Times, March 12, 1981

 “Although the statistical association does not prove
 that coffee causes cancer, Dr. Brian MacMahon of
 Harvard, leader of the research group, said he stopped
 drinking coffee a few months ago when the results of
 the study became clear. In a telephone interview, he
 said that he would not presume to advise others.”
Or Get A Refill
Or Get A Refill
CRITICS SAY COFFEE STUDY WAS FLAWED
New York Times, June 30, 1981

 ''This otherwise excellent paper may be flawed in one
 critical way,'' said a letter from Dr. Steven Shedlofsky of
 the Veterans Administration Hospital in White River
 Junction, Vt. He questioned the comparison of
 pancreatic cancer patients with persons hospitalized
 for noncancerous diseases of the digestive system.
Or Get A Refill
CRITICS SAY COFFEE STUDY WAS FLAWED
New York Times, June 30, 1981

 “Such patients, he noted, might be expected to give up
 coffee drinking because of their illness. This, he argued,
 would tilt the proportion of coffee drinkers away from
 the ''control'' group who were being compared with
 the cancer patients. Amplifying the letter in an
 interview, Dr. Shedlofsky said many patients with
 digestive diseases give up coffee because they believe
 it aggravates their discomfort, and others do so
 because their doctors have advised them to.
We Cured Cancer 15 Years Ago
We Cured Cancer 15 Years Ago
The New York Times, May 3, 1998

 “Within a year, if all goes well, the first cancer patient will
 be injected with two new drugs that can eradicate any
 type of cancer, with no obvious side effects and no drug
 resistance -- in mice.”
 …
 ''Judah is going to cure cancer in two years,'' said Dr.
 James D. Watson, a Nobel laureate who directs the Cold
 Spring Harbor Laboratory, a cancer research center on
 Long Island. Dr. Watson said Dr. Folkman would be
 remembered along with scientists like Charles Darwin as
 someone who permanently altered civilization.”
Or Maybe We Didn’t. Here’s Why.
The New York Times,
February 11, 2013
Or Maybe We Didn’t. Here’s Why.
The New York Times, February 11, 2013

   “The study’s findings do not mean that mice are useless
   models for all human diseases. But, its authors said,
   they do raise troubling questions about diseases like
   the ones in the study that involve the immune system,
   including cancer and heart disease.”
Do You Like Being Wrong?
5,000 compounds started out for the market
How many made it to clinical (human) trials?
Do You Like Being Wrong?
5,000 compounds started out for the market
How many made it to clinical (human) trials?
                   5
How many of those made it to FDA approval?
Do You Like Being Wrong?
5,000 compounds started out for the market
How many made it to clinical (human) trials?
                   5
How many of those made it to FDA approval?
                   1

     Source: http://www.phrma.org/issues/intellectual-property (PhRMA)
How to Get It Wrong
• Write about compounds in pre-clinical trials as
  if they were about to be on pharmacy shelves
• Write about every drug in phase I and phase II
  trials as if it would definitely be approved
How Often Are Studies Wrong?




               Ioannidis JPA. PLoS Med 2005; 2(8): e124
How Often Are Studies Wrong?
Retractions on the Rise




                 -The Wall Street Journal
“But My Editor Wants More Copy”
      So you’ll be covering studies.
How Not to Get It Wrong




Source: Jamie Smith http://inksnow.blogspot.com/2012/07/context.html
Always Read the Study
Always Read the Study



Get the full study and read it –
“I think it’s journalistic malpractice to not have the full
study in front of you when you’re reporting,” Oransky
says.
How to Get Studies
• www.EurekAlert.org for embargoed material
• AHCJ membership includes access to Cochrane
  Library, Health Affairs, JAMA, and many other
  journals www.healthjournalism.org
• ScienceDirect (Elsevier) gives reporters free access
  to hundreds of journals www.sciencedirect.com
• Open access journals (e.g., Public Library of Science
  www.plos.org)
• Ask press officers, or the authors
Ask “Dumb” Questions


If you lack experience dealing with scientific material,
don’t be afraid to ask for definitions of jargon and
scientific terms. This is no time to pretend you
understand everything. Oransky says the science and
medical industries are full of jargon that mask important
details. “You’ll get off the phone and have a notebook
full of gibberish and jargon,” he says. “You can’t be
afraid of asking a dumb question.”
Ask Smart Questions
• Was it:
  – Peer-reviewed?
  – Published? Where? Not all journals are
    created equal.

   “Dr. X said they published in Y rather than a
     clinical journal because the paper was too
     long for the word limits in the clinical
     journals. I'm not sure where a detail like that
     would go…but he was impressed with my
     question.”
Ask Smart Questions
• Was it in humans?
  – It’s remarkable there are any mice left with
    cancer, depression, or restless leg syndrome
Ask Smart Questions
• Size matters
Look for the power calculation, and ask if you
  don’t see one
Ask Smart
    Questions
• Was it well-designed?




                          From Covering Medical Research, Schwitzer/AHCJ
Ask Smart Questions
• “Were those your primary endpoints?”
• “Looks as though that endpoint reached statistical
  significance. Is that difference clinically
  significant?”
Read the Discussion
Good journals will insist that authors include
  limitations.
Read accompanying editorials, too.
What’s Your Angle?
• Are you trying to help readers, listeners, and
  viewers make better health care decisions?
• Covering a study because it has a good business
  angle, or it’s about a local project, is perfectly OK,
  but it doesn’t mean readers deserve less
  evidence and skepticism
Who Could Benefit?
• How many people have the disease?
• Keep potential disease-mongering in mind
How Effective is the Treatment?
• Clinically significant endpoints, or surrogates –
  does this matter?
• Preventing complications? How many?
• Always remember to quantify results, not just
  “patients improved”
What Are The Side Effects?
• Every treatment has them
• Where to look:
  – Go beyond press releases and abstracts
  – Look at tables, charts, and results sections
Who Dropped Out?
• Why did they leave the trial?
• Intention to treat analysis
How Much Does it Cost?
• If it’s ready to be the subject of a story,
  someone has projected the likely cost and
  market.
   – At least ask.
Who Has an Interest?
• Disclose conflicts
• PharmedOut.org
• Dollars For Docs series
  http://projects.propublica.org/docdollars/
Are There Alternatives?
• Did the study compare the new treatment to
  existing alternatives, or to placebo?
• What are the advantages and disadvantages
  (and costs) of those existing alternatives?
• Consider alternative explanations. Remember
  coffee and pancreatic cancer?
Don’t Rely Only on Study Authors
• Find outside sources. Here’s how:
Use Anecdotes Carefully
• Is the story representative?
• Does the source of the story have any conflicts?
Watch Your Language
• Lifestyle/diet – are they randomized controlled
  trials, or just observational?
• If observational, make the language fit the
  evidence:
   – YES: “tied,” “linked”
   – NO: “reduces,” “causes”
A Dirty Little Secret




Keep a biostatistician in your back pocket

                               Photo by Peyri Herrera, on Flickr
Keep Yourself Honest




• Use HealthNewsReview.org
Acknowledgement/Contact
• Nancy Lapid, Reuters Health

          ivan-oransky@erols.com
           Twitter: @ivanoransky

More Related Content

What's hot

The Retraction Watch Database
The Retraction Watch DatabaseThe Retraction Watch Database
The Retraction Watch DatabaseIvan Oransky
 
Lessons from 1,889 story reviews
Lessons from 1,889 story reviewsLessons from 1,889 story reviews
Lessons from 1,889 story reviewsGary Schwitzer
 
Pfizer vice president blows the whistle
Pfizer vice president blows the whistlePfizer vice president blows the whistle
Pfizer vice president blows the whistlekirti betai
 
Debunking Made Easy, AHCJ 2014
Debunking Made Easy, AHCJ 2014Debunking Made Easy, AHCJ 2014
Debunking Made Easy, AHCJ 2014Ivan Oransky
 
Role of the media in preventing or promoting overdiagnosis
Role of the media in preventing or promoting overdiagnosisRole of the media in preventing or promoting overdiagnosis
Role of the media in preventing or promoting overdiagnosisGary Schwitzer
 
How Do We Know What We Know?
How Do We Know What We Know?How Do We Know What We Know?
How Do We Know What We Know?Ivan Oransky
 
Fake Peer Review: What We've Learned at Retraction Watch
Fake Peer Review: What We've Learned at Retraction WatchFake Peer Review: What We've Learned at Retraction Watch
Fake Peer Review: What We've Learned at Retraction WatchIvan Oransky
 
Fixing Science: The Replicability Crisis
Fixing Science: The Replicability CrisisFixing Science: The Replicability Crisis
Fixing Science: The Replicability CrisisAlex Holcombe
 
Scrutiny of Science Goes Public
Scrutiny of Science Goes PublicScrutiny of Science Goes Public
Scrutiny of Science Goes PublicIvan Oransky
 
Reproducibility, open access, open science
Reproducibility, open access, open scienceReproducibility, open access, open science
Reproducibility, open access, open scienceAlex Holcombe
 
Scott Edmunds for #PeerRevWk17: Confessions of a pre-print & open peer review...
Scott Edmunds for #PeerRevWk17: Confessions of a pre-print & open peer review...Scott Edmunds for #PeerRevWk17: Confessions of a pre-print & open peer review...
Scott Edmunds for #PeerRevWk17: Confessions of a pre-print & open peer review...GigaScience, BGI Hong Kong
 
Advice to junior researchers: High or low road to success?
Advice to junior researchers: High or low road to success?Advice to junior researchers: High or low road to success?
Advice to junior researchers: High or low road to success?James Coyne
 
APRIN talk, Tokyo, March 2018
APRIN talk, Tokyo, March 2018APRIN talk, Tokyo, March 2018
APRIN talk, Tokyo, March 2018Ivan Oransky
 
What Is Peer Review?
What Is Peer Review?What Is Peer Review?
What Is Peer Review?Jerry Stovall
 
Problem-citations--CrossrefLive18--2018-11-13
Problem-citations--CrossrefLive18--2018-11-13Problem-citations--CrossrefLive18--2018-11-13
Problem-citations--CrossrefLive18--2018-11-13jodischneider
 
Continued citation of bad science and what we can do about it--2021-04-20
Continued citation of bad science and what we can do about it--2021-04-20Continued citation of bad science and what we can do about it--2021-04-20
Continued citation of bad science and what we can do about it--2021-04-20jodischneider
 
Introduction to Peer review, updated 2015-03-05
Introduction to Peer review, updated 2015-03-05Introduction to Peer review, updated 2015-03-05
Introduction to Peer review, updated 2015-03-05Wouter Gerritsma
 

What's hot (20)

The Retraction Watch Database
The Retraction Watch DatabaseThe Retraction Watch Database
The Retraction Watch Database
 
Lessons from 1,889 story reviews
Lessons from 1,889 story reviewsLessons from 1,889 story reviews
Lessons from 1,889 story reviews
 
Pfizer vice president blows the whistle
Pfizer vice president blows the whistlePfizer vice president blows the whistle
Pfizer vice president blows the whistle
 
Debunking Made Easy, AHCJ 2014
Debunking Made Easy, AHCJ 2014Debunking Made Easy, AHCJ 2014
Debunking Made Easy, AHCJ 2014
 
Role of the media in preventing or promoting overdiagnosis
Role of the media in preventing or promoting overdiagnosisRole of the media in preventing or promoting overdiagnosis
Role of the media in preventing or promoting overdiagnosis
 
How Do We Know What We Know?
How Do We Know What We Know?How Do We Know What We Know?
How Do We Know What We Know?
 
Fake Peer Review: What We've Learned at Retraction Watch
Fake Peer Review: What We've Learned at Retraction WatchFake Peer Review: What We've Learned at Retraction Watch
Fake Peer Review: What We've Learned at Retraction Watch
 
Fixing Science: The Replicability Crisis
Fixing Science: The Replicability CrisisFixing Science: The Replicability Crisis
Fixing Science: The Replicability Crisis
 
Scrutiny of Science Goes Public
Scrutiny of Science Goes PublicScrutiny of Science Goes Public
Scrutiny of Science Goes Public
 
ECCVID 2020
ECCVID 2020ECCVID 2020
ECCVID 2020
 
Reproducibility, open access, open science
Reproducibility, open access, open scienceReproducibility, open access, open science
Reproducibility, open access, open science
 
Scott Edmunds for #PeerRevWk17: Confessions of a pre-print & open peer review...
Scott Edmunds for #PeerRevWk17: Confessions of a pre-print & open peer review...Scott Edmunds for #PeerRevWk17: Confessions of a pre-print & open peer review...
Scott Edmunds for #PeerRevWk17: Confessions of a pre-print & open peer review...
 
Advice to junior researchers: High or low road to success?
Advice to junior researchers: High or low road to success?Advice to junior researchers: High or low road to success?
Advice to junior researchers: High or low road to success?
 
APRIN talk, Tokyo, March 2018
APRIN talk, Tokyo, March 2018APRIN talk, Tokyo, March 2018
APRIN talk, Tokyo, March 2018
 
What Is Peer Review?
What Is Peer Review?What Is Peer Review?
What Is Peer Review?
 
Problem-citations--CrossrefLive18--2018-11-13
Problem-citations--CrossrefLive18--2018-11-13Problem-citations--CrossrefLive18--2018-11-13
Problem-citations--CrossrefLive18--2018-11-13
 
ofsp.10693
ofsp.10693ofsp.10693
ofsp.10693
 
Continued citation of bad science and what we can do about it--2021-04-20
Continued citation of bad science and what we can do about it--2021-04-20Continued citation of bad science and what we can do about it--2021-04-20
Continued citation of bad science and what we can do about it--2021-04-20
 
Introduction to Peer review, updated 2015-03-05
Introduction to Peer review, updated 2015-03-05Introduction to Peer review, updated 2015-03-05
Introduction to Peer review, updated 2015-03-05
 
Suzanne M. Rivera and Heide Aungst "What Specimen Donors Think (and Considera...
Suzanne M. Rivera and Heide Aungst "What Specimen Donors Think (and Considera...Suzanne M. Rivera and Heide Aungst "What Specimen Donors Think (and Considera...
Suzanne M. Rivera and Heide Aungst "What Specimen Donors Think (and Considera...
 

Viewers also liked

Васильківській міській організації ветеранів України 30 років
Васильківській міській організації ветеранів України 30 роківВасильківській міській організації ветеранів України 30 років
Васильківській міській організації ветеранів України 30 роківТайны города Василькова
 
TALLER TRABAJADO EN CLASE♥
TALLER TRABAJADO EN CLASE♥TALLER TRABAJADO EN CLASE♥
TALLER TRABAJADO EN CLASE♥yekaortiz
 
Southwest Airlines Case Brief
Southwest Airlines Case BriefSouthwest Airlines Case Brief
Southwest Airlines Case BriefIrtaza Ishtiaq
 
Proyectos Estudiantiles DI - Bases
Proyectos Estudiantiles DI - BasesProyectos Estudiantiles DI - Bases
Proyectos Estudiantiles DI - Basescee_info_2012
 
CPD training module_naeem tariq
CPD training module_naeem tariqCPD training module_naeem tariq
CPD training module_naeem tariqNaeem Tariq
 
HubSpot State of Inbound 2014
HubSpot State of Inbound 2014HubSpot State of Inbound 2014
HubSpot State of Inbound 2014Dan St. Peter
 
CES 471 - FINAL PAPER - electronic
CES 471 - FINAL PAPER - electronicCES 471 - FINAL PAPER - electronic
CES 471 - FINAL PAPER - electronicMitchell Schutte
 
Simplifying the Complex: Serving Data from Pipeline Data Models
Simplifying the Complex: Serving Data from Pipeline Data ModelsSimplifying the Complex: Serving Data from Pipeline Data Models
Simplifying the Complex: Serving Data from Pipeline Data ModelsSafe Software
 
Mind map title sequnace ideas
Mind map title sequnace ideasMind map title sequnace ideas
Mind map title sequnace ideasJazmine Sims
 
Evaluacion metodologica de la pregunta etnico racial del censo 2011 de costa ...
Evaluacion metodologica de la pregunta etnico racial del censo 2011 de costa ...Evaluacion metodologica de la pregunta etnico racial del censo 2011 de costa ...
Evaluacion metodologica de la pregunta etnico racial del censo 2011 de costa ...Escuela Electoral y de Gobernabilidad
 
Diseña tu visión personal
Diseña tu visión personalDiseña tu visión personal
Diseña tu visión personalcosabuhh
 
Web 2.0: It's All about Social Networking
Web 2.0: It's All about Social NetworkingWeb 2.0: It's All about Social Networking
Web 2.0: It's All about Social Networkingeoshea
 
Constitución mexicana
Constitución mexicanaConstitución mexicana
Constitución mexicanaGeera Bahena
 

Viewers also liked (20)

Pgu slide yerasyl2
Pgu slide yerasyl2Pgu slide yerasyl2
Pgu slide yerasyl2
 
Васильківській міській організації ветеранів України 30 років
Васильківській міській організації ветеранів України 30 роківВасильківській міській організації ветеранів України 30 років
Васильківській міській організації ветеранів України 30 років
 
Por las avenidas del modelo educativo...
Por las avenidas del modelo educativo...Por las avenidas del modelo educativo...
Por las avenidas del modelo educativo...
 
TALLER TRABAJADO EN CLASE♥
TALLER TRABAJADO EN CLASE♥TALLER TRABAJADO EN CLASE♥
TALLER TRABAJADO EN CLASE♥
 
Southwest Airlines Case Brief
Southwest Airlines Case BriefSouthwest Airlines Case Brief
Southwest Airlines Case Brief
 
Proyectos Estudiantiles DI - Bases
Proyectos Estudiantiles DI - BasesProyectos Estudiantiles DI - Bases
Proyectos Estudiantiles DI - Bases
 
CPD training module_naeem tariq
CPD training module_naeem tariqCPD training module_naeem tariq
CPD training module_naeem tariq
 
HubSpot State of Inbound 2014
HubSpot State of Inbound 2014HubSpot State of Inbound 2014
HubSpot State of Inbound 2014
 
3. BPW
3. BPW3. BPW
3. BPW
 
CES 471 - FINAL PAPER - electronic
CES 471 - FINAL PAPER - electronicCES 471 - FINAL PAPER - electronic
CES 471 - FINAL PAPER - electronic
 
Simplifying the Complex: Serving Data from Pipeline Data Models
Simplifying the Complex: Serving Data from Pipeline Data ModelsSimplifying the Complex: Serving Data from Pipeline Data Models
Simplifying the Complex: Serving Data from Pipeline Data Models
 
Presentación de caso clínico
Presentación de caso clínicoPresentación de caso clínico
Presentación de caso clínico
 
Cuerda
CuerdaCuerda
Cuerda
 
Mind map title sequnace ideas
Mind map title sequnace ideasMind map title sequnace ideas
Mind map title sequnace ideas
 
Evaluacion metodologica de la pregunta etnico racial del censo 2011 de costa ...
Evaluacion metodologica de la pregunta etnico racial del censo 2011 de costa ...Evaluacion metodologica de la pregunta etnico racial del censo 2011 de costa ...
Evaluacion metodologica de la pregunta etnico racial del censo 2011 de costa ...
 
Diseña tu visión personal
Diseña tu visión personalDiseña tu visión personal
Diseña tu visión personal
 
Web 2.0: It's All about Social Networking
Web 2.0: It's All about Social NetworkingWeb 2.0: It's All about Social Networking
Web 2.0: It's All about Social Networking
 
Constitución mexicana
Constitución mexicanaConstitución mexicana
Constitución mexicana
 
Brief imagen publica diciembre 2016
Brief imagen publica diciembre 2016Brief imagen publica diciembre 2016
Brief imagen publica diciembre 2016
 
Magicnature
MagicnatureMagicnature
Magicnature
 

Similar to Covering Medical Studies: How Not to Get It Wrong

Evaluating medical evidence for journalists
Evaluating medical evidence for journalistsEvaluating medical evidence for journalists
Evaluating medical evidence for journalistsIvan Oransky
 
Covering Screening Tests: Do No Harm (As A Reporter)
Covering Screening Tests: Do No Harm (As A Reporter)Covering Screening Tests: Do No Harm (As A Reporter)
Covering Screening Tests: Do No Harm (As A Reporter)Ivan Oransky
 
Schwitzer keynote to ISDM 2013 Lima, Peru
Schwitzer keynote to ISDM 2013 Lima, PeruSchwitzer keynote to ISDM 2013 Lima, Peru
Schwitzer keynote to ISDM 2013 Lima, PeruGary Schwitzer
 
McGovern Award Lecture - American Medical Writers Association
McGovern Award Lecture - American Medical Writers AssociationMcGovern Award Lecture - American Medical Writers Association
McGovern Award Lecture - American Medical Writers AssociationGary Schwitzer
 
The scandal of the £5m PACE chronic fatigue trial
The scandal of the £5m PACE chronic fatigue trialThe scandal of the £5m PACE chronic fatigue trial
The scandal of the £5m PACE chronic fatigue trialJames Coyne
 
The scandal of the £5m pace trial for myalgic encephalomyelitis
The scandal of the £5m pace trial for myalgic encephalomyelitisThe scandal of the £5m pace trial for myalgic encephalomyelitis
The scandal of the £5m pace trial for myalgic encephalomyelitisJames Coyne
 
1. What are the science community’s two standards for considering .docx
1. What are the science community’s two standards for considering .docx1. What are the science community’s two standards for considering .docx
1. What are the science community’s two standards for considering .docxjeremylockett77
 
MIT Medical Evidence Bootcamp for Journalists 2012
MIT Medical Evidence Bootcamp for Journalists 2012MIT Medical Evidence Bootcamp for Journalists 2012
MIT Medical Evidence Bootcamp for Journalists 2012GarySchwitzer
 
Covering Cancer Treatments: Tricks of the Trade
Covering Cancer Treatments: Tricks of the TradeCovering Cancer Treatments: Tricks of the Trade
Covering Cancer Treatments: Tricks of the TradeIvan Oransky
 
Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...
Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...
Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...Gary Schwitzer
 
Research Fundamentals for Activists
Research Fundamentals for ActivistsResearch Fundamentals for Activists
Research Fundamentals for ActivistsHopkinsCFAR
 
Family Med Orientation July 2009
Family Med Orientation July 2009Family Med Orientation July 2009
Family Med Orientation July 2009Robin Featherstone
 
Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013
Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013
Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013Gary Schwitzer
 
Introduction to Evidence Based Medicine
Introduction to Evidence Based MedicineIntroduction to Evidence Based Medicine
Introduction to Evidence Based MedicinePaul Albert
 
Evaluating Information
Evaluating InformationEvaluating Information
Evaluating Informationbibliotecaag
 
MedicalResearch.com: Medical Research Exclusive Interviews July 24 2015
MedicalResearch.com:  Medical Research Exclusive Interviews July 24 2015MedicalResearch.com:  Medical Research Exclusive Interviews July 24 2015
MedicalResearch.com: Medical Research Exclusive Interviews July 24 2015Marie Benz MD FAAD
 
Lessons to improve our reporting on evidence - analysis of 1,600 stories over...
Lessons to improve our reporting on evidence - analysis of 1,600 stories over...Lessons to improve our reporting on evidence - analysis of 1,600 stories over...
Lessons to improve our reporting on evidence - analysis of 1,600 stories over...Gary Schwitzer
 
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...Gary Schwitzer
 
Angrist Fogm vi_7march2013
Angrist Fogm vi_7march2013Angrist Fogm vi_7march2013
Angrist Fogm vi_7march2013Misha Angrist
 
Why we review health care news releases
Why we review health care news releasesWhy we review health care news releases
Why we review health care news releasesKevin Lomangino
 

Similar to Covering Medical Studies: How Not to Get It Wrong (20)

Evaluating medical evidence for journalists
Evaluating medical evidence for journalistsEvaluating medical evidence for journalists
Evaluating medical evidence for journalists
 
Covering Screening Tests: Do No Harm (As A Reporter)
Covering Screening Tests: Do No Harm (As A Reporter)Covering Screening Tests: Do No Harm (As A Reporter)
Covering Screening Tests: Do No Harm (As A Reporter)
 
Schwitzer keynote to ISDM 2013 Lima, Peru
Schwitzer keynote to ISDM 2013 Lima, PeruSchwitzer keynote to ISDM 2013 Lima, Peru
Schwitzer keynote to ISDM 2013 Lima, Peru
 
McGovern Award Lecture - American Medical Writers Association
McGovern Award Lecture - American Medical Writers AssociationMcGovern Award Lecture - American Medical Writers Association
McGovern Award Lecture - American Medical Writers Association
 
The scandal of the £5m PACE chronic fatigue trial
The scandal of the £5m PACE chronic fatigue trialThe scandal of the £5m PACE chronic fatigue trial
The scandal of the £5m PACE chronic fatigue trial
 
The scandal of the £5m pace trial for myalgic encephalomyelitis
The scandal of the £5m pace trial for myalgic encephalomyelitisThe scandal of the £5m pace trial for myalgic encephalomyelitis
The scandal of the £5m pace trial for myalgic encephalomyelitis
 
1. What are the science community’s two standards for considering .docx
1. What are the science community’s two standards for considering .docx1. What are the science community’s two standards for considering .docx
1. What are the science community’s two standards for considering .docx
 
MIT Medical Evidence Bootcamp for Journalists 2012
MIT Medical Evidence Bootcamp for Journalists 2012MIT Medical Evidence Bootcamp for Journalists 2012
MIT Medical Evidence Bootcamp for Journalists 2012
 
Covering Cancer Treatments: Tricks of the Trade
Covering Cancer Treatments: Tricks of the TradeCovering Cancer Treatments: Tricks of the Trade
Covering Cancer Treatments: Tricks of the Trade
 
Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...
Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...
Bioethics lecture UMDNJ-RWJ Medical School: "Addressing the Morass at the Int...
 
Research Fundamentals for Activists
Research Fundamentals for ActivistsResearch Fundamentals for Activists
Research Fundamentals for Activists
 
Family Med Orientation July 2009
Family Med Orientation July 2009Family Med Orientation July 2009
Family Med Orientation July 2009
 
Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013
Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013
Knight Science Journalism Fellowships at MIT Medical Evidence Boot Camp 2013
 
Introduction to Evidence Based Medicine
Introduction to Evidence Based MedicineIntroduction to Evidence Based Medicine
Introduction to Evidence Based Medicine
 
Evaluating Information
Evaluating InformationEvaluating Information
Evaluating Information
 
MedicalResearch.com: Medical Research Exclusive Interviews July 24 2015
MedicalResearch.com:  Medical Research Exclusive Interviews July 24 2015MedicalResearch.com:  Medical Research Exclusive Interviews July 24 2015
MedicalResearch.com: Medical Research Exclusive Interviews July 24 2015
 
Lessons to improve our reporting on evidence - analysis of 1,600 stories over...
Lessons to improve our reporting on evidence - analysis of 1,600 stories over...Lessons to improve our reporting on evidence - analysis of 1,600 stories over...
Lessons to improve our reporting on evidence - analysis of 1,600 stories over...
 
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...
Association of Health Care Journalists workshop 2015, Santa Clara, Schwitzer ...
 
Angrist Fogm vi_7march2013
Angrist Fogm vi_7march2013Angrist Fogm vi_7march2013
Angrist Fogm vi_7march2013
 
Why we review health care news releases
Why we review health care news releasesWhy we review health care news releases
Why we review health care news releases
 

More from Ivan Oransky

The relationship between PubPeer comments and the speed of retraction
The relationship between PubPeer comments and the speed of retractionThe relationship between PubPeer comments and the speed of retraction
The relationship between PubPeer comments and the speed of retractionIvan Oransky
 
During COVID-19, Are Preprints The Problem?
During COVID-19, Are Preprints The Problem?During COVID-19, Are Preprints The Problem?
During COVID-19, Are Preprints The Problem?Ivan Oransky
 
Does Science Self-Correct?
Does Science Self-Correct?Does Science Self-Correct?
Does Science Self-Correct?Ivan Oransky
 
Why We Should Be Talking About Reproducibility -- But Not Forget About Fraud
Why We Should Be Talking About Reproducibility -- But Not Forget About FraudWhy We Should Be Talking About Reproducibility -- But Not Forget About Fraud
Why We Should Be Talking About Reproducibility -- But Not Forget About FraudIvan Oransky
 
WCRI 2019: What makes a story a story?
WCRI 2019: What makes a story a story?WCRI 2019: What makes a story a story?
WCRI 2019: What makes a story a story?Ivan Oransky
 
FNLM Carlisle talk
FNLM Carlisle talkFNLM Carlisle talk
FNLM Carlisle talkIvan Oransky
 
Does Social Psychology Really Have More Retractions?
Does Social Psychology Really Have More Retractions?Does Social Psychology Really Have More Retractions?
Does Social Psychology Really Have More Retractions?Ivan Oransky
 
How to pitch reporters without being annoying
How to pitch reporters without being annoyingHow to pitch reporters without being annoying
How to pitch reporters without being annoyingIvan Oransky
 
Does Stem Cell Research Have A Retraction Problem?
Does Stem Cell Research Have A Retraction Problem?Does Stem Cell Research Have A Retraction Problem?
Does Stem Cell Research Have A Retraction Problem?Ivan Oransky
 
Coverage of Clinical Medicine: A Diagnosis and Treatment Plan
Coverage of Clinical Medicine: A Diagnosis and Treatment PlanCoverage of Clinical Medicine: A Diagnosis and Treatment Plan
Coverage of Clinical Medicine: A Diagnosis and Treatment PlanIvan Oransky
 
Shoot The Messenger? Challenges in Medical Journalism
Shoot The Messenger? Challenges in Medical JournalismShoot The Messenger? Challenges in Medical Journalism
Shoot The Messenger? Challenges in Medical JournalismIvan Oransky
 
Retractions, Peer Review, and Transparency
Retractions, Peer Review, and TransparencyRetractions, Peer Review, and Transparency
Retractions, Peer Review, and TransparencyIvan Oransky
 
Post-Publication Peer Review in Science: Reflections on Retractions and Med...
Post-Publication Peer Review in Science: Reflections on Retractions and Med...Post-Publication Peer Review in Science: Reflections on Retractions and Med...
Post-Publication Peer Review in Science: Reflections on Retractions and Med...Ivan Oransky
 
New Media, Traditional Media: How are messages getting through in today’s eco...
New Media, Traditional Media: How are messages getting through in today’s eco...New Media, Traditional Media: How are messages getting through in today’s eco...
New Media, Traditional Media: How are messages getting through in today’s eco...Ivan Oransky
 
Covering Medical Studies: Tips and Tricks (my NABJ presentation)
Covering Medical Studies: Tips and Tricks (my NABJ presentation)Covering Medical Studies: Tips and Tricks (my NABJ presentation)
Covering Medical Studies: Tips and Tricks (my NABJ presentation)Ivan Oransky
 
Can We Still Trust Science?
Can We Still Trust Science?Can We Still Trust Science?
Can We Still Trust Science?Ivan Oransky
 
Fnlm june 10 2013 final
Fnlm june 10 2013 finalFnlm june 10 2013 final
Fnlm june 10 2013 finalIvan Oransky
 
Can You Trust What You Read In (Scientific and News)papers?
Can You Trust What You Read In (Scientific and News)papers?Can You Trust What You Read In (Scientific and News)papers?
Can You Trust What You Read In (Scientific and News)papers?Ivan Oransky
 
We Are All Gatekeepers Now: Scrutiny of Science Goes Public
We Are All Gatekeepers Now: Scrutiny of Science Goes PublicWe Are All Gatekeepers Now: Scrutiny of Science Goes Public
We Are All Gatekeepers Now: Scrutiny of Science Goes PublicIvan Oransky
 

More from Ivan Oransky (20)

The relationship between PubPeer comments and the speed of retraction
The relationship between PubPeer comments and the speed of retractionThe relationship between PubPeer comments and the speed of retraction
The relationship between PubPeer comments and the speed of retraction
 
During COVID-19, Are Preprints The Problem?
During COVID-19, Are Preprints The Problem?During COVID-19, Are Preprints The Problem?
During COVID-19, Are Preprints The Problem?
 
Does Science Self-Correct?
Does Science Self-Correct?Does Science Self-Correct?
Does Science Self-Correct?
 
Why We Should Be Talking About Reproducibility -- But Not Forget About Fraud
Why We Should Be Talking About Reproducibility -- But Not Forget About FraudWhy We Should Be Talking About Reproducibility -- But Not Forget About Fraud
Why We Should Be Talking About Reproducibility -- But Not Forget About Fraud
 
WCRI 2019: What makes a story a story?
WCRI 2019: What makes a story a story?WCRI 2019: What makes a story a story?
WCRI 2019: What makes a story a story?
 
FNLM Carlisle talk
FNLM Carlisle talkFNLM Carlisle talk
FNLM Carlisle talk
 
Does Social Psychology Really Have More Retractions?
Does Social Psychology Really Have More Retractions?Does Social Psychology Really Have More Retractions?
Does Social Psychology Really Have More Retractions?
 
How to pitch reporters without being annoying
How to pitch reporters without being annoyingHow to pitch reporters without being annoying
How to pitch reporters without being annoying
 
Does Stem Cell Research Have A Retraction Problem?
Does Stem Cell Research Have A Retraction Problem?Does Stem Cell Research Have A Retraction Problem?
Does Stem Cell Research Have A Retraction Problem?
 
Rutgers
RutgersRutgers
Rutgers
 
Coverage of Clinical Medicine: A Diagnosis and Treatment Plan
Coverage of Clinical Medicine: A Diagnosis and Treatment PlanCoverage of Clinical Medicine: A Diagnosis and Treatment Plan
Coverage of Clinical Medicine: A Diagnosis and Treatment Plan
 
Shoot The Messenger? Challenges in Medical Journalism
Shoot The Messenger? Challenges in Medical JournalismShoot The Messenger? Challenges in Medical Journalism
Shoot The Messenger? Challenges in Medical Journalism
 
Retractions, Peer Review, and Transparency
Retractions, Peer Review, and TransparencyRetractions, Peer Review, and Transparency
Retractions, Peer Review, and Transparency
 
Post-Publication Peer Review in Science: Reflections on Retractions and Med...
Post-Publication Peer Review in Science: Reflections on Retractions and Med...Post-Publication Peer Review in Science: Reflections on Retractions and Med...
Post-Publication Peer Review in Science: Reflections on Retractions and Med...
 
New Media, Traditional Media: How are messages getting through in today’s eco...
New Media, Traditional Media: How are messages getting through in today’s eco...New Media, Traditional Media: How are messages getting through in today’s eco...
New Media, Traditional Media: How are messages getting through in today’s eco...
 
Covering Medical Studies: Tips and Tricks (my NABJ presentation)
Covering Medical Studies: Tips and Tricks (my NABJ presentation)Covering Medical Studies: Tips and Tricks (my NABJ presentation)
Covering Medical Studies: Tips and Tricks (my NABJ presentation)
 
Can We Still Trust Science?
Can We Still Trust Science?Can We Still Trust Science?
Can We Still Trust Science?
 
Fnlm june 10 2013 final
Fnlm june 10 2013 finalFnlm june 10 2013 final
Fnlm june 10 2013 final
 
Can You Trust What You Read In (Scientific and News)papers?
Can You Trust What You Read In (Scientific and News)papers?Can You Trust What You Read In (Scientific and News)papers?
Can You Trust What You Read In (Scientific and News)papers?
 
We Are All Gatekeepers Now: Scrutiny of Science Goes Public
We Are All Gatekeepers Now: Scrutiny of Science Goes PublicWe Are All Gatekeepers Now: Scrutiny of Science Goes Public
We Are All Gatekeepers Now: Scrutiny of Science Goes Public
 

Covering Medical Studies: How Not to Get It Wrong

  • 1. Covering Medical Studies: How Not to Get It Wrong AHCJ Boston, March 2013 Ivan Oransky, MD Executive Editor, Reuters Health Co-Founder, Retraction Watch @ivanoransky
  • 3. Put Down That Coffee!
  • 4. Put Down That Coffee! STUDY LINKS COFFEE USE TO PANCREAS CANCER New York Times, March 12, 1981 “Although the statistical association does not prove that coffee causes cancer, Dr. Brian MacMahon of Harvard, leader of the research group, said he stopped drinking coffee a few months ago when the results of the study became clear. In a telephone interview, he said that he would not presume to advise others.”
  • 5. Or Get A Refill
  • 6. Or Get A Refill CRITICS SAY COFFEE STUDY WAS FLAWED New York Times, June 30, 1981 ''This otherwise excellent paper may be flawed in one critical way,'' said a letter from Dr. Steven Shedlofsky of the Veterans Administration Hospital in White River Junction, Vt. He questioned the comparison of pancreatic cancer patients with persons hospitalized for noncancerous diseases of the digestive system.
  • 7. Or Get A Refill CRITICS SAY COFFEE STUDY WAS FLAWED New York Times, June 30, 1981 “Such patients, he noted, might be expected to give up coffee drinking because of their illness. This, he argued, would tilt the proportion of coffee drinkers away from the ''control'' group who were being compared with the cancer patients. Amplifying the letter in an interview, Dr. Shedlofsky said many patients with digestive diseases give up coffee because they believe it aggravates their discomfort, and others do so because their doctors have advised them to.
  • 8. We Cured Cancer 15 Years Ago
  • 9. We Cured Cancer 15 Years Ago The New York Times, May 3, 1998 “Within a year, if all goes well, the first cancer patient will be injected with two new drugs that can eradicate any type of cancer, with no obvious side effects and no drug resistance -- in mice.” … ''Judah is going to cure cancer in two years,'' said Dr. James D. Watson, a Nobel laureate who directs the Cold Spring Harbor Laboratory, a cancer research center on Long Island. Dr. Watson said Dr. Folkman would be remembered along with scientists like Charles Darwin as someone who permanently altered civilization.”
  • 10. Or Maybe We Didn’t. Here’s Why. The New York Times, February 11, 2013
  • 11. Or Maybe We Didn’t. Here’s Why. The New York Times, February 11, 2013 “The study’s findings do not mean that mice are useless models for all human diseases. But, its authors said, they do raise troubling questions about diseases like the ones in the study that involve the immune system, including cancer and heart disease.”
  • 12. Do You Like Being Wrong? 5,000 compounds started out for the market How many made it to clinical (human) trials?
  • 13. Do You Like Being Wrong? 5,000 compounds started out for the market How many made it to clinical (human) trials? 5 How many of those made it to FDA approval?
  • 14. Do You Like Being Wrong? 5,000 compounds started out for the market How many made it to clinical (human) trials? 5 How many of those made it to FDA approval? 1 Source: http://www.phrma.org/issues/intellectual-property (PhRMA)
  • 15. How to Get It Wrong • Write about compounds in pre-clinical trials as if they were about to be on pharmacy shelves • Write about every drug in phase I and phase II trials as if it would definitely be approved
  • 16. How Often Are Studies Wrong? Ioannidis JPA. PLoS Med 2005; 2(8): e124
  • 17. How Often Are Studies Wrong?
  • 18. Retractions on the Rise -The Wall Street Journal
  • 19. “But My Editor Wants More Copy” So you’ll be covering studies.
  • 20. How Not to Get It Wrong Source: Jamie Smith http://inksnow.blogspot.com/2012/07/context.html
  • 22. Always Read the Study Get the full study and read it – “I think it’s journalistic malpractice to not have the full study in front of you when you’re reporting,” Oransky says.
  • 23. How to Get Studies • www.EurekAlert.org for embargoed material • AHCJ membership includes access to Cochrane Library, Health Affairs, JAMA, and many other journals www.healthjournalism.org • ScienceDirect (Elsevier) gives reporters free access to hundreds of journals www.sciencedirect.com • Open access journals (e.g., Public Library of Science www.plos.org) • Ask press officers, or the authors
  • 24. Ask “Dumb” Questions If you lack experience dealing with scientific material, don’t be afraid to ask for definitions of jargon and scientific terms. This is no time to pretend you understand everything. Oransky says the science and medical industries are full of jargon that mask important details. “You’ll get off the phone and have a notebook full of gibberish and jargon,” he says. “You can’t be afraid of asking a dumb question.”
  • 25. Ask Smart Questions • Was it: – Peer-reviewed? – Published? Where? Not all journals are created equal. “Dr. X said they published in Y rather than a clinical journal because the paper was too long for the word limits in the clinical journals. I'm not sure where a detail like that would go…but he was impressed with my question.”
  • 26. Ask Smart Questions • Was it in humans? – It’s remarkable there are any mice left with cancer, depression, or restless leg syndrome
  • 27. Ask Smart Questions • Size matters Look for the power calculation, and ask if you don’t see one
  • 28. Ask Smart Questions • Was it well-designed? From Covering Medical Research, Schwitzer/AHCJ
  • 29. Ask Smart Questions • “Were those your primary endpoints?” • “Looks as though that endpoint reached statistical significance. Is that difference clinically significant?”
  • 30. Read the Discussion Good journals will insist that authors include limitations. Read accompanying editorials, too.
  • 31. What’s Your Angle? • Are you trying to help readers, listeners, and viewers make better health care decisions? • Covering a study because it has a good business angle, or it’s about a local project, is perfectly OK, but it doesn’t mean readers deserve less evidence and skepticism
  • 32. Who Could Benefit? • How many people have the disease? • Keep potential disease-mongering in mind
  • 33. How Effective is the Treatment? • Clinically significant endpoints, or surrogates – does this matter? • Preventing complications? How many? • Always remember to quantify results, not just “patients improved”
  • 34. What Are The Side Effects? • Every treatment has them • Where to look: – Go beyond press releases and abstracts – Look at tables, charts, and results sections
  • 35. Who Dropped Out? • Why did they leave the trial? • Intention to treat analysis
  • 36. How Much Does it Cost? • If it’s ready to be the subject of a story, someone has projected the likely cost and market. – At least ask.
  • 37. Who Has an Interest? • Disclose conflicts • PharmedOut.org • Dollars For Docs series http://projects.propublica.org/docdollars/
  • 38. Are There Alternatives? • Did the study compare the new treatment to existing alternatives, or to placebo? • What are the advantages and disadvantages (and costs) of those existing alternatives? • Consider alternative explanations. Remember coffee and pancreatic cancer?
  • 39. Don’t Rely Only on Study Authors • Find outside sources. Here’s how:
  • 40. Use Anecdotes Carefully • Is the story representative? • Does the source of the story have any conflicts?
  • 41. Watch Your Language • Lifestyle/diet – are they randomized controlled trials, or just observational? • If observational, make the language fit the evidence: – YES: “tied,” “linked” – NO: “reduces,” “causes”
  • 42. A Dirty Little Secret Keep a biostatistician in your back pocket Photo by Peyri Herrera, on Flickr
  • 43. Keep Yourself Honest • Use HealthNewsReview.org
  • 44. Acknowledgement/Contact • Nancy Lapid, Reuters Health ivan-oransky@erols.com Twitter: @ivanoransky