Many researchers have studied the relationship between diet and health. Specifically, there are papers showing an association between the consumption of sugar sweetened beverages and Type 2 diabetes. Many meta-analyses use individual studies that do not attempt to adjust for multiple testing or multiple modeling. Hence the claims reported in a meta-analysis paper may be unreliable as the base papers do not ensure unbiased statistics. Georgia Southern researchers obtained copies of each of the 10 papers used in a meta-analysis paper and counted the numbers of outcomes, predictors, and covariates. They then estimated the size of the potential analysis search space available to the authors of these papers; i.e. the number of comparisons and models available. The potential analysis search space is the number of outcomes times the number of predictors times 2c, where c is the number of covariates. This formula was applied to information found in the abstracts (Space A) as well as the text (Space T) of each base paper. Researchers found that: the median and range of the number of comparisons possible across the base papers are 6.5 and (2 to 12,288), respectively for Space A, and 196,608 and (3072 to 117,117,952), respectively for Space T. It is noted that the median of 6.5 for Space A may be misleading as each study has 60-165 foods that could be predictors. We concluded that since testing is at the 5 percent level and the number of comparisons is very large, nominal statistical significance is very weak support for a claim. The claims in these papers are not statistically supported and hence are unreliable, therefore conclusions In the meta-analysis paper is also unreliable.
See also: PubMed, https://www.ncbi.nlm.nih.gov/pubmed/30465718
Authors are Dr. Karl Peace, Dr. Jingjing Yin, Dr. Haresh Rochani, faculty at Georgia Southern University Jiann-Ping Hsu Collage of Public Health (JPHCOPH), Mr. Sarbesh Pandeya, graduate student, JPHCOPH and Dr. Stanley Young of CGSTAT, Raleigh, NC, adjunct professor in biostatistics, JPHCOPH/ GSU.