A Wellness Perspective On Scientific Studies: Be Skeptical, and When Religious Studies Are Involved, Be Even More Skeptical
Donald B. Ardell, Ph.D.
According to medical scholar and epidemiologist John Ioannidis, any given research claim is more likely to be erroneous than accurate.1 Since research is a $55-billion-a-year business, you might think the validity of new study findings would be investigated by the media before any research claims were heralded to the world via banner headlines. But that rarely happens. Instead, we read daily of amazing findings, only to discover later that the findings are not as exciting, promising or positive as first expressed. Given the sensationalism of these stories, it's a wonder we have not all become more guarded, even jaded, about news of the latest studies. But we have not. Too often, "discoveries" are viewed as if they were new and revised editions of the medieval bible, with all the crazy stuff edited out and genuinely sensible guides for peace on earth added for good measure.
-
More Parenting Analysis:
- Religion and Child Development by Dr. Jean Mercer
- Religious Benefits to Children Unproven by Amanda Metskas
There are many reasons we should be disposed not to believe any given new study finding. Here is a summary of the kinds of methodological and other shortcomings found in most studies, as fully described in Dr. Ioannidis' 2005 essay "Why Most Published Research Findings Are False." In this report, Ioannidis emphasizes that flawed findings stem not from fraud or other misconduct, but from poor study design, self-serving data analysis and/or miscalculations.
Researchers often tinker too much with the statistical variables of their analysis to coax meaningful insights from the data sets. Too little attention is given to the replication of published scientific results. Research findings are too rarely checked by others. Retractions are still relatively infrequent. Findings that have been refuted can linger in the scientific literature for years to be cited unwittingly by other researchers, compounding the errors. And it's difficult to distinguish error from fraud, sloppiness from deception, eagerness from greed.
The bottom line as a result is that any given research claim is more likely to be false than true.
It's even harder to separate scientific conviction from partisan passion. As scientific findings become fodder for political policy wars over matters from stem-cell research to global warming, even trivial errors and corrections can have larger consequences.
The latter problem seems especially pronounced in a long-term, generously-funded project called The National Study of Youth and Religion (NSYR). Let's count the possible issues here. No one of these is necessarily damning, but collectively, bells begin to ring: The co-director teaches at a religious university; funding comes from the John Templeton Foundation, a group passionately dedicated to proving that religion is "good for you." But a bigger concern is the stated purpose of the project: "To enhance our understanding of the religious lives of American youth"! Does that not make you a little uneasy about the objectivity of this endeavor? What will the investigators do if the data suggests religious dogma causes zits? Will they then be able to separate scientific conviction from partisan passion?
Consider some of the other stated purposes of the study:
- To identify effective religious practices.
- To describe the perceived effectiveness of religious communities to their youth.
- To foster an informed discussion about the influence of religion in order to encourage sustained reflection about... youth and religion
I submit that such purposes, not to mention such sponsorship, while suitable for an advocacy group like Pat Robertson's 700 Club, are inappropriate, dishonest and manipulative when embraced by social scientists who should be devoted to conducting objective investigations.
Maybe I'm being too hard on these devoted scholars, overly influenced by my own skepticism about studies in general and about studies by religious people on religious issues in particular. Maybe I am too much of a secular humanist, given as I am to free inquiry, reason, empiricism and rationality.
But one other alarm bell I got from perusing the website of NSYR is that they are none too anxious to suffer skeptics like me poking around looking for bias or other methodological flaws. A site message provides the following stern admonition: "We strongly suggest that anyone interested in analyzing the NSYR data invest time studying all of the supporting documentation provided and in reading through the NSYR methods report." Furthermore, for those out there thinking of questioning any of this, be advised: "The NSYR survey data set is complex. Conducting analysis using the data requires familiarity with the survey instrument, the variables, the data formatting and the general study methodology." Translation: Give up your day job if necessary in order to devote hours if not days of study of our work before you dare ask questions. I'm hard-pressed to think of another scientific study that has couched itself in such pay-no-attention-to-that-man-behind-the-curtain warnings.
I discussed this with my sometime editor Dale McGowan, who observed, "The fact that they go on and on about how thoroughly one must prepare before commenting on their work is a red flag to me – something like theologians who insist that one must read Aquinas in the original Latin before criticizing Trinitarianism."
I decided that, considering the criteria set out by the NSYR project before anyone should dare attempt a critique of this work, I'd give it a pass, since I could not bear to spend quite that much time on religion and youth. Surely they were counting on that.
However, I might change my mind when the project findings are released. Yes, when the investigators conclude that religion makes young people warmer in Winter, cooler in Summer and helps them sleep better all year round, that religious youth are invariably fun, romantic and hip, sexy and free, have higher morale and more antibodies to resist disease, pay lower insurance rates, are tax exempt and get better gas mileage than kids with no faith in supernatural beings—I might reconsider. All this may be true, and no doubt the media will splash feature headlines like "Religion Rocks." Nevertheless, remember the reservations noted in Dr. Ioannidis' essay and moderate your enthusiasm.
1 Text and abstract available at pubmedcentral.nih.gov