The Problem With Multiple Comparisons, Coursework Example

The problem of multiple comparisons is particularly important in the field of public health because researchers are trying to devise evidence-based interventions based on real patterns in the data. Put simply, the problem of multiple comparisons refers to the phenomenon that in any data set, a number of questions may be ruled “significant” (p<.05) via statistical hypothesis testing. When the number of questions asked of a particular data set notably increase, however, it becomes more difficult to distinguish which questions have a truly significant p-value and which are significant by random chance.

A provocative epidemiological study by Matthews et al. entitled “You are what your mother eats: evidence for maternal preconception diet influencing foetal sex in humans” illustrated this problem. The authors claimed that selection of breakfast foods by pregnant women could influence the gender of the baby by finding significant results when they tested 132 different foods over two time periods: a total of 264 statistical tests (Mathews, Johnson, & Neil, 2008). As Young et al. point out, however, nearly all of the study’s “statistical” findings are a function of the sheer number of questions asked and consistent with pure chance as evidenced by a p-value plot (Young, Bang, &Oktay, 2009). That is, multiple comparisons can lead to false positives.

Data fishing is a phenomenon when a researcher does not a prioristate a hypothesis regarding what he/she is looking for in the data; rather, he/she finds patterns in the data and post hoc advocates a hypothesis based on those patterns.  Data fishing essentially subverts the scientific process by using existing analytical techniques to explain patterns in the data rather than independently test them.  As a result, data fishing renders p-values meaningless and the inability to delineate between real differences and differences due to random change in the data.

Dahl et al. show how data fishing may lead to a false conclusion.  While looking at the data shows that smoking may be a strong predictor of lower back pain, in reality, the relationship is on the borderline of significance and needs further evaluation. In order to prevent data fishing, the researcher should cite his/her hypothesis and biological mechanism for purported results before engaging in data analysis.

Citations

Dahl, F.A., Grotle, M., Benth, J.S., Natvig, B. (2008).  Data splitting as a countermeasure against hypothesis fishing: with a case of study predictors for low back pain.  European Journal of Epidemiology. 23, 237-242. Doi:10.1007/s10654-008-9230.

Mathews F., Johnson P.J., &Neil A. (2008). You are what your mother eats: evidence for maternal preconception diet influencing foetal sex in humans. Proceedings of the Royal Society. 275 (1661), 1668. Doi:10.1098/rspb.2008.0105.

Young, S.S., Bang, H., &Oktay, K. (2009).Cereal-induced gender selection? Most likely multiple testing false positive.  Proceedings of the Royal Society in Biological Sciences.276(1660), 1211-1212.