Ongoing technological developments have made it easier than ever before for scientists to share their data, materials, and analysis code. Sharing data and analysis code makes it easier for other researchers to re-use or check published research. These benefits will only emerge if researchers can reproduce the analysis reported in published articles, and if data is annotated well enough so that it is clear what all variables mean. Because most researchers have not been trained in computational reproducibility, it is important to evaluate current practices to identify practices that can be improved. We examined data and code sharing, as well as computational reproducibility of the main results, without contacting the original authors, for Registered Reports published in the psychological literature between 2014 and 2018. Of the 62 articles that met our inclusion criteria, data was available for 40 articles, and analysis scripts for 37 articles. For the 35 articles that shared both data and code and performed analyses in SPSS, R, Python, MATLAB, or JASP, we could run the scripts for 31 articles, and reproduce the main results for 20 articles. Although the articles that shared both data and code (35 out of 62, or 56%) and articles that could be computationally reproduced (20 out of 35, or 57%) was relatively high compared to other studies, there is clear room for improvement. We provide practical recommendations based on our observations, and link to examples of good research practices in the papers we reproduced.

Registered Reports: Peer review before results are known to align scientific values and practices.

Registered Reports is a publishing format used by over 250 journals that emphasizes the importance of the research question and the quality of methodology by conducting peer review prior to data collection. High quality protocols are then provisionally accepted for publication if the authors follow through with the registered methodology.

This format is designed to reward best practices in adhering to the hypothetico-deductive model of the scientific method. It eliminates a variety of questionable research practices, including low statistical power, selective reporting of results, and publication bias, while allowing complete flexibility to report serendipitous findings.

This page includes information on Registered Reports including readings on Registered Reports, Participating Journals, Details & Workflow, Resources for Editors, Resources For Funders, FAQs, and Allied Initiatives.

In this deep dive session, Amanda Montoya (UCLA) and Karen Rambo-Hernandez (Texas A&M University) introduce the basics of preregistration and Registered Reports: two methods for creating a permanent record of a research plan prior to conducting data collection. They discuss the conceptual similarities and practical differences between pre-registration and registered reports. They provide practical advice from their own experiences using these practices in research labs and resources available for researchers interested in using these approaches. The session concludes with questions and discussion about adopting these practices and unique considerations for implementing these practices in education research.

Deep Dive on Open Practices: Understanding Registered Reports in Education Research with Amanda Montoya and Betsy McCoach - Registered reports are a new publication mechanism where peer review and the decision to publish the results of a study occur prior to data collection and/or analysis. Registered reports share many characteristics with preregistration but are distinct by involving the journal prior to completing the study. Journals in the field of education are increasingly offering opportunities to publish registered reports. Registered reports offer a variety of benefits to both the researcher and to the research field. In this workshop, we will discuss the basics of registered reports, benefits and limitations of registered reports, and which journals in education accept registered reports. We provide some practical advice on deciding which projects are appropriate for registered reports, implementing registered reports, and time management throughout the process. We discuss how special cases can be implemented as registered reports, such as secondary data analysis, replications, meta-analyses, and longitudinal studies.

Registered Reports (RRs) is a publishing model in which initial peer review is conducted prior to knowing the outcomes of the research. In-principle acceptance of papers at this review stage combats publication bias, and provides a clear distinction between confirmatory and exploratory research. Some editors raise a practical concern about adopting RRs. By reducing publication bias, RRs may produce more negative or mixed results and, if such results are not valued by the research community, receive less citations as a consequence. If so, by adopting RRs, a journal’s impact factor may decline. Despite known flaws with impact factor, it is still used as a heuristic for judging journal prestige and quality. Whatever the merits of considering impact factor as a decision-rule for adopting RRs, it is worthwhile to know whether RRs are cited less than other articles. We will conduct a naturalistic comparison of citation and altmetric impact between published RRs and comparable empirical articles from the same journals.

Registered reports present a substantial departure from traditional publishing models with the goal of enhancing the transparency and credibility of the scientific literature. We map the evolving universe of registered reports to assess their growth, implementation and shortcomings at journals across scientific disciplines.

Hear from Andrew Foster, editor at the Journal of Development Economics, and Irenaeus Wolff, a guest editor for Experimental Economics, as they discuss their experiences with implementing the Registered Reports format, how it was received by authors, and the trends they noticed after adoption. Aleksandar Bogdanoski of BITSS also joins us to explore pre-results review, how to facilitate the process at journals, and best practices for supporting authors and reviewers.

This webinar addresses questions related to writing, reviewing, editing, or funding a study using the Registered Report format, featuring Chris Chambers and ...

The recent ‘replication crisis’ in psychology has focused attention on ways of increasing methodological rigor within the behavioral sciences. Part of this work has involved promoting ‘Registered Reports’, wherein journals peer review papers prior to data collection and publication. Although this approach is usually seen as a relatively recent development, we note that a prototype of this publishing model was initiated in the mid-1970s by parapsychologist Martin Johnson in the European Journal of Parapsychology (EJP). A retrospective and observational comparison of Registered and non-Registered Reports published in the EJP during a seventeen-year period provides circumstantial evidence to suggest that the approach helped to reduce questionable research practices. This paper aims both to bring Johnson’s pioneering work to a wider audience, and to investigate the positive role that Registered Reports may play in helping to promote higher methodological and statistical standards.