Reproducibility of scientific findings has long been an important indicator of the validity of data gleaned from research. At the Sackler Colloquium focused on “Reproducibility of Research: Issues and Proposed Remedies,” Dean David B. Allison, Indiana University School of Public Health – Bloomington, and co-organizers Drs. Richard Shiffrin, Victoria Stodden and the late Stephen Fienberg invited discussion on defining reproducibility in various research contexts and providing remedies that contribute to greater reproducibility and transparency.
[Photo: Dean David B. Allison, Dr. Andrew Brown, Dr. Richard Shiffrin]
A dozen articles shaping the colloquium were published in the March 2018 special issue of Proceedings of the National Academy of Sciences of the United States of America. Featured among them is “Scientific Progress Despite Irreproducibility: A Seeming Paradox,” led by Dr. Shiffrin. It seems contradictory that the way science is practiced makes quite a few reported findings difficult or impossible to reproduce. However, scientific practice has evolved and advanced despite elemental problems, such as that years may have to pass before the validity, importance and usability of findings can be ascertained. It is possible for invalid findings and conclusions to be reproduced, but the repetition of fundamental — and frequently avoidable — errors is usually at the core. The researchers recommend a balance between sharing of promising early-stage results and maintaining of strict vigilance to ensure quality reporting. Shared data must be reliable and important, but also of true scientific value. They examine proposed new remedies designed to reduce the degree of irreproducibility and recommend reforms tailored to research type and goals. They caution, however, that any remedy and reform can have unintended consequences that can slow rather than speed scientific progress.
Also highlighted is a report headed by Dr. Andrew Brown, assistant professor in applied health science at IU-Bloomington, titled “Issues with Data and Analyses: Errors, Underlying Themes, and Potential Solutions,” which focuses on errors that could have been avoided by application of good established practices. The researchers discuss such influences as citation and publication bias, mathematical miscalculations, errors in interpretation and working with bad data that was obtained through questionable methods, designs or techniques. Incorrect management and storage of data can also affect the ability to confirm findings, as can errors of communication and logic. They stress that consequences of invalid interpretation and reporting range from the loss of public trust to the potential loss of life. The merits and drawbacks of the peer review process as a reliable upholder of scientific integrity in the literature are also deliberated. Dr. Brown and the team maintain that there are certain actions and methods that well-trained scientists should and could recognize as erroneous and/or lacking in rigor.
Notes Dr. Brown, “We were interested in identifying challenges and limitations but, more importantly, in identifying paths to make science more rigorous and reproducible. Work needs to be done to determine underlying causes, more rigorously catalog errors and pitfalls, and identify the best interventions for reinforcing scientific rigor. There are incremental and structural changes to make to improve the present condition of scientific investigations and publications.”
Videos of these talks and other colloquium presentations can be viewed on the Sackler YouTube Channel.