Fraud and Why Studies are Flawed: Should Journalists Trust Peer Review?
•Download as PPT, PDF•
2 likes•1,751 views
Talk for the Scripps Howard Institute on the Environment and Science at Florida Atlantic University, May 23, 2012
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Fraud and Why Studies are Flawed: Should Journalists Trust Peer Review?
Similar to Fraud and Why Studies are Flawed: Should Journalists Trust Peer Review? (20)
More from Ivan Oransky
More from Ivan Oransky (20)
Recently uploaded
Recently uploaded (10)
Fraud and Why Studies are Flawed: Should Journalists Trust Peer Review?
- 1. Fraud and Why Studies are Flawed: Should Journalists Trust Peer Review? Ivan Oransky, MD Co-Founder, Retraction Watch Executive Editor, Reuters Health Scripps Howard Institute on the Environment and Science May 23, 2012
- 2. Can You Trust Journal Studies? • How good is peer review? • Positive publication bias • Over-reliance on embargoed studies • How often it turns out to be wrong • How to get it right
- 3. How Does Peer Review Work? • Usually three outside reviewers • Usually anonymous • Sometimes in as little as 72 hours
- 4. How Good Is Peer Review?
- 5. How Good Is Peer Review?
- 6. How Good Is Peer Review? • Bad at detecting fraud • Slow • Costly (and unfunded) • Prone to bias
- 7. Peer Reviewers: Worse With Experience
- 9. Positive Publication Bias Publish a trial that will bring US$100,000 of profit or meet the end-of-year budget by firing an editor. -- Former BMJ editor Richard Smith
- 10. Embargoes and the Ingelfinger Rule By the late 20th century, journals needed to compete not just with each other but with newspapers and other media…In 1969, the Journal articulated this relationship in its Ingelfinger Rule, a policy against publishing anything that had already appeared elsewhere. Other journals followed suit. This rule, combined with embargo policies, has led to a carefully choreographed production in which medical journals and the popular press work cooperatively and competitively to influence the news cycle. -- NEJM, April 19, 2012
- 11. Even Without Embargoes, We’d Still Have Ingelfinger
- 12. How Often Are Studies Wrong? Ioannidis JPA. PLoS Med 2005; 2(8): e124
- 13. How Often Are Studies Wrong?
- 14. Retraction Watch • http://retractionwatch.com • Launched August 2010 with Adam Marcus • Frequently cited in major news outlets, from Nature to Der Spiegel to New York Times • 250,000 pageviews per month
- 15. Retractions on the Rise -The Wall Street Journal
- 16. Retractions on the Rise
- 17. Retractions on the Rise -Neil Saunders
- 18. The Unofficial Record Holder
- 19. Why Do Journals Retract? -Nature
- 20. Why Do Journals Retract? • Error is more common than fraud • 73.5% of papers retracted for error (or undisclosed reason) vs 26.6% for fraud • Most common cause of retraction: scientific mistake (234 papers; 31.5%) • Fabrication (including data plagiarism) more common than text plagiarism • 67 retractions (9.0%) had multiple causes, but 134 papers (18.1%) retracted for ambiguous reasons -Journal of Medical Ethics 2010
- 23. Is Fraud on the Rise? Over the years…surveys have asked scientists directly about their behaviour…. [T]hese studies have used different methods and asked different questions, so their results have been deemed inconclusive and/or difficult to compare. A non- systematic review based on survey and non-survey data led to estimate that the frequency of “serious misconduct”, including plagiarism, is near 1%. -- Fanelli, PLoS ONE, 2009
- 24. Is Fraud on the Rise? A pooled weighted average of 1.97% (N = 7, 95%CI: 0.86–4.45) of scientists admitted to have fabricated, falsified or modified data or results at least once – a serious form of misconduct by any standard – and up to 33.7% admitted other questionable research practices. In surveys asking about the behaviour of colleagues, admission rates were 14.12% (N = 12, 95% CI: 9.91–19.72) for falsification, and up to 72% for other questionable research practices. -- Fanelli, PLoS ONE, 2009
- 25. Is Fraud on the Rise? Meta-regression showed that self reports surveys, surveys using the words “falsification” or “fabrication”, and mailed surveys yielded lower percentages of misconduct. When these factors were controlled for, misconduct was reported more frequently by medical/pharmacological researchers than others. -- Fanelli, PLoS ONE, 2009
- 26. Is Fraud on the Rise? Considering that these surveys ask sensitive questions and have other limitations, it appears likely that this is a conservative estimate of the true prevalence of scientific misconduct. -- Fanelli, PLoS ONE, 2009
- 28. Should Reporters Cover Conferences?
- 29. Conference Pitfalls • Conferences select presenters based on < 1000 words • Urologists at U of Florida & Indiana U studied 126 randomized controlled trials presented in 2002-2003
- 30. Conference Pitfalls • RCTs are the “gold standard” of medical evidence • But the quality of that evidence wasn’t pretty • No abstract said how trial subjects were randomly assigned to different treatments or placebos • None told how the study ensured that neither the researchers nor their doctors knew which they got • Only about a quarter said how long researchers followed the subjects in the trial
- 31. Just Say No Sometimes, it’s better not to cover something. But if you must…
- 32. Always Read the Study Writing about a study after reading just a press release or an abstract – without reading the entire paper – is journalistic malpractice
- 33. How to Get Studies • www.EurekAlert.org for embargoed material • Association of Health Care Journalists membership includes access to Cochrane Library, Health Affairs, JAMA, and many other journals www.healthjournalism.org • ScienceDirect (Elsevier) gives reporters free access to hundreds of journals www.sciencedirect.com • Open access journals (e.g., Public Library of Science www.plos.org) • Ask press officers, or the authors
- 34. Who Has an Interest? • Disclose conflicts • PharmedOut.org • Dollars For Docs series http://projects.propublica.org/docdollars/
- 35. Don’t Rely Only on Study Authors • Find outside sources. Here’s how:
- 36. A Dirty Little Secret Keep a biostatistician in your back pocket Photo by Peyri Herrera, on Flickr
- 37. Acknowledgement/Contact Thanks: Nancy Lapid, Reuters Health http://retractionwatch.com ivan-oransky@erols.com Twitter: @ivanoransky