Building Consensus on Community Standards for Reproducible Science

Abstract:

As geochemists, the traditional model by which standard methods for generating, presenting, and using data have been generated relied on input from the community, the results of seminal studies, a variety of authoritative bodies, and has required a great deal of time. The rate of technological and related policy change has accelerated to the point that this historical model does not satisfy the needs of the community, publishers, or funders.

The development of a new mechanism for building consensus raises a number of questions:

• Which aspects of our data are the focus of reproducibility standards?
• Who sets the standards?
• How do we subdivide the development of the consensus? 

We propose an open, transparent, and inclusive approach to the development of data and reproducibility standards that is organized around specific sub-disciplines and driven by the community of practitioners in those sub-disciplines. It should involve editors, program managers, and representatives of domain data facilities as well as professional societies, but avoid any single group to be the final authority. A successful example of this model is the Editors Roundtable, a cross section of editors, funders, and data facility managers that discussed and agreed on leading practices for the reporting of geochemical data in publications, including accessibility and format of the data, data quality information, and metadata and identifiers for samples (Goldstein et al., 2014). We argue that development of data and reproducibility standards needs to heavily rely on representatives from the community of practitioners to set priorities and provide perspective.

Groups of editors, practicing scientists, and other stakeholders would be assigned the task of reviewing existing practices and recommending changes as deemed necessary. They would weigh the costs and benefits of changing the standards for that community, propose appropriate tools to facilitate those changes, work through the professional societies, gain community support, collect suggestions/edits, and ultimately approval and implementation through the journals. Domain data facilities such as the Interdisciplinary Earth Data Alliance (IEDA) can facilitate this process and support the groups.

Goldstein et al. (2014): EarthChem Library. http://dx.doi.org/10.1594/IEDA/100426