A recent journal article in the Review of Religious Research claims that the negative outcomes typically observed in research on pornography are really just a self-fulfilling prophecy. In the article, Idaho State University sociologist Jeremy Thomas and his colleagues conclude that, if your church tells you that viewing pornography harms you or your marriage, then it will harm you. Pornography is harmful not because it is inherently dangerous, but because you believe it’s harmful.
The piece, “Anti-Pornography Narratives as Self-Fulfilling Prophecies,” sets out to demonstrate “how the anti-pornography narratives that are predominant within different religious traditions can influence the effect that pornography viewing has on the marital happiness of husbands within those traditions.” But this claim rests on a slew of methodological and statistical errors that make Thomas’s conclusions dubious at best.
Thomas discusses three pornography narratives: traditional values, public-performer harm, and personal-viewer harm, with emphasis on the harm pornography does to the individual viewer. To test the prevalence of these among religious groups, he examined keyword usage in three religious magazines, one representing Evangelicals (Christianity Today), another representing mainline Protestantism (The Christian Century), and a third representing liberal Catholicism (Commonweal) and compared the outcomes to those of three secular magazines (Time, Newsweek, and U.S. News & World Report). Thomas found that keywords invoking the personal-view harm narrative occurred most frequently in Christianity Today (18.7 percent of paragraphs examined), compared to 13.8 and 9.7 percent for The Christian Century and Commonweal, respectively. Comparable numbers were between 10.7 and 11.8 percent for the secular magazines.
Thomas then moves on to an analysis of the General Social Survey to examine if Evangelical, mainline Protestant, or liberal Catholic husbands have lower marital happiness when they view pornography. The way Thomas established whether husbands viewed pornography was by asking, “Have you seen an X-rated movie the past year?” Besides the fact that the X-designation no longer exists, circumstances for watching pornography matter greatly; movies are generally viewed for entertainment rather than sexual purposes. Yet Thomas pushes past this critical validity issue and examines whether viewing pornography (again, defined as watching “an X-rated movie”) is associated with marital happiness.
Oddly, Thomas throws out much of the information on his dependent variable of marital happiness by collapsing a three-category variable (very happy, pretty happy, not too happy) into two categories (very happy vs. everyone else), a decision for which he offers no justification. Researchers almost never do this because it limits the claims one can make. He finds that pornography viewing is negatively correlated with being “very happy,” but only for Evangelicals. Among mainline Protestants and liberal Catholics, he found no evidence that pornography was associated with husbands’ marital happiness. In his final model – where he includes all husbands rather than limiting the model to each religious tradition – he finds that husbands who view pornography are less likely to be “very happy” with their marriages than those who do not, but finds no evidence of religious differences in marital happiness. In the final block, he examines whether the effect differs by religious group by including an interaction term – a statistical technique used to see if the effect of one variable on another differs across groups – and finds, again, no evidence of religious differences.
Yet, this is not what he tells the reader. Instead, he claims that, for interaction terms, “we cannot rely on p-values [the traditional statistical tool for measuring whether an effect is likely to be real or not] alone to determine statistical significance; rather interaction terms must be evaluated in a graphical manner in order to assess their statistical significance.” He then alleges to demonstrate (via graphs) that the effect is statistically significant for Evangelicals, suggesting that pornography’s harms are more pronounced among Evangelicals.
Let’s back up just a bit. Thomas wants to know two separate pieces of information. First, whether there is an association between viewing pornography and marital happiness and, second, whether that association – if it exists – works the same across religious groups (i.e., whether pornography is equally destructive to marriages across all groups or more pronounced for one of them). Because there are two different pieces of information, statisticians use two different tests: simple slope for the first piece and interaction terms for the second. Unfortunately, Thomas conflates the two tests. The test he uses “in a graphical manner… to assess their statistical significance” can show only the first piece of information: higher levels of pornography viewing are linked to lower marital happiness for Evangelicals but not for non-Evangelicals (the remaining graphs show no statistical associations). For the second piece of information, we must use the interaction term, which tells us that — though pornography viewing is statistically significantly linked to less marital happiness among Evangelicals — the size of the effect is not actually different from everyone else, meaning the association between pornography viewing and marital happiness works exactly the same across religious groups. In other words, Thomas doesn’t find a bigger effect in Evangelicals, only clearer evidence of the same effect that he found in the other groups. His claims are simply not supported by his own data.
On top of this critical misinterpretation, Thomas’s paper suffers from other methodological and statistical flaws. For instance, he doesn’t control for how long people have been married. While marital duration may seem irrelevant to this particular topic, the academic literature has consistently shown that it’s one of the most important factors in evaluating marital happiness and quality; disregarding it entirely is an enormous omission. Second, Thomas says he employs “standardized binary logistic regression coefficients” for his religious tradition variables. This is nonsensical. In statistics, standardized variables are used to compare variables measured on different scales; say, years of education vs. dollars of income. Instead, Thomas coded religious tradition in a simple are-you-or-aren’t-you fashion. Besides, Thomas appears to use typical log-odds, standard fare for binary logistic regression, rather than standardized binary logistic regression coefficients. Third, Thomas deletes GSS respondents who failed to answer relevant questions. Though seemingly practical, this “listwise deletion” is among the most biased ways to handle data, and far inferior to the preferred method of modeling patterns of missing data that might reveal a different story.
All of these flaws leave statistically-savvy readers questioning the trustworthiness of the results and the paper. Thomas finishes by claiming he and his colleagues found evidence of variation in the influence of viewing pornography on marital happiness. In fact, his data show no evidence of variations between different Christian denominations, though viewing pornography is linked with lower marital happiness across the board.
That’s not exactly new… nor groundbreaking.