Scientific leaders from government, industry, academia and publishing identified and proposed solutions to one the most complex challenges facing scientists today – reproducibility in research – at the recent Hood College – Frederick National Laboratory Life Sciences Symposium.
“Research reproducibility has been in the spotlight for many years, and while some progress has been made, it’s a very complex issue with multiple layers that require many solutions, rather than a one size fits all fix,” said Leonard Freedman, Ph.D., chief science officer at the Frederick National Laboratory for Cancer Research, who led the symposium and development of the Scientific Standards Hub.
Nearly 200 participants joined for the Reproducibility in Science Symposium that was held June 16-18 at Hood College. This year's biennial interdisciplinary conference in the Hood College-Frederick National Laboratory Life Sciences Symposium series focused on elevating standards to strengthen research.
“It’s a very important topic, … and it really matched to the highest expectations,” said attendee Manpreet Singh, Ph.D., a computational research scientist at The Henry M. Jackson Foundation for the Advancement of Military Medicine, in support of the Biotechnology High Performance Computing Software Applications Institute.
Several presentations showcased tools and plans for improving reproducibility: a pageant ranging from data repositories to assay manuals to genomic and proteomic techniques. There was a palpable interest among the audience.
“I’m going to take back to our lab a lot of the information and try to build upon it,” said Gabriel Benton, Ph.D., a quality control scientist in the Frederick National Laboratory's Applied and Developmental Research Directorate.
A first step: What is reproducibility?
Keynote speaker Harvey Fineberg, M.D., Ph.D., professor emeritus of Health Policy and Management at the Harvard T. H. Chan School of Public Health, said just defining reproducibility is challenging, as disciplines view it differently.
The National Science Foundation says reproducibility is “the ability of a researcher to duplicate the results of a prior study using the same materials and procedures used by the original investigator.”
Fineberg proposed a blanket definition with distinct subcategories. The structure establishes a shared vocabulary and divides the overarching issue into clearer portions that scientists can more easily address.
Some attendees cautioned their peers against equating reproducibility with scientific validity, urging them to monitor their work for biases. Remaining clear-eyed would foster reproduction of sound data, they said.
Others reminded attendees that irreproducibility is more often due to extenuating factors or misinterpreted data than fraud. Multiple attendees advocated for better outreach to the public to build confidence in science.
“The scientific community needs to really listen to what the general public thinks about the need for data reproducibility,” said attendee Thomas “TC” D.Y. Chung, Ph.D., director of Translational Programs Outreach at the Conrad Prebys Center for Chemical Genomics.
Document the work in detail and publish negative data
Given their diverse scientific fields, attendees later broke into smaller groups to allow for in-depth discussions among leading stakeholders. The topics included research antibody characterization, recombinant protein production, preclinical drug discovery, organoids and in vitro New Approach Methodologies (NAMs), liquid biopsies and AI standards.
The proteins group came up with a three-phase plan to establish common standards and a protein repository. The committee on AI outlined best practices for wider adoption. Others made comparable progress.
In the preclinical drug studies group, moderator Nathan Coussens, Ph.D., director of the Frederick National Laboratory's In Vitro Evaluation and Molecular Pharmacology Laboratories, asked the participants to identify areas for improvement. Nearly half indicated laboratory protocols, data analysis, and data reporting.
“What are the minimum details someone needs to reproduce [protocols]?” one panelist asked, adding scientists should record small steps they might normally omit.
One panelist likened published protocols to “a cooking recipe.” They skip some details, like an assay’s pH conditions, under the assumption the reader knows or will work in similar conditions.
Another panelist said his well-documented “bulletproof” assay failed when executed by his postdoctoral fellow. He discovered an undocumented, seemingly inconsequential factor—the amount of time the samples remained at room temperature—was the culprit.
Around the room, ideas sprouted from the exchange.
One participant suggested curating and promoting repositories where scientists can publish their full protocols, which are often too extensive for scientific journals.
Incentivize scientists to publish negative data, their experiments that don’t work, another participant said. This would tell others not to waste their time on the failed hypothesis. It also reduces data biases caused by the abundance of published successful studies.
“Maybe bioRxiv would be a way for people to not only publish their negative data but get credit for it,” he said.
The symposium focused on finding solutions to bolster reproducibility in research rather than pointing blame or criticism at the many ways it fails to occur, Freedman said, and noted that the topic resonated with the nearly 200 highly engaged participants.
Media Inquiries
Mary Ellen Hackett
Manager, Communications Office
301-401-8670