Updating search results...

Search Resources

4 Results

View
Selected filters:
Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking
Unrestricted Use
CC BY
Rating
0.0 stars

The designing, collecting, analyzing, and reporting of psychological studies entail many choices that are often arbitrary. The opportunistic use of these so-called researcher degrees of freedom aimed at obtaining statistically significant results is problematic because it enhances the chances of false positive results and may inflate effect size estimates. In this review article, we present an extensive list of 34 degrees of freedom that researchers have in formulating hypotheses, and in designing, running, analyzing, and reporting of psychological research. The list can be used in research methods education, and as a checklist to assess the quality of preregistrations and to determine the potential for bias due to (arbitrary) choices in unregistered studies.

Subject:
Psychology
Social Science
Material Type:
Reading
Provider:
Frontiers in Psychology
Author:
Coosje L. S. Veldkamp
Hilde E. M. Augusteijn
Jelte M. Wicherts
Marcel A. L. M. van Assen
Marjan Bakker
Robbie C. M. van Aert
Date Added:
08/07/2020
Secondary Data Preregistration
Unrestricted Use
Public Domain
Rating
0.0 stars

Preregistration is the process of specifying project details, such as hypotheses, data collection procedures, and analytical decisions, prior to conducting a study. It is designed to make a clearer distinction between data-driven, exploratory work and a-priori, confirmatory work. Both modes of research are valuable, but are easy to unintentionally conflate. See the Preregistration Revolution for more background and recommendations.

For research that uses existing datasets, there is an increased risk of analysts being biased by preliminary trends in the dataset. However, that risk can be balanced by proper blinding to any summary statistics in the dataset and the use of hold out datasets (where the "training" and "validation" datasets are kept separate from each other). See this page for specific recommendations about "split samples" or "hold out" datasets. Finally, if those procedures are not followed, disclosure of possible biases can inform the researcher and her audience about the proper role any results should have (i.e. the results should be deemed mostly exploratory and ideal for additional confirmation).

This project contains a template for creating your preregistration, designed specifically for research using existing data. In the future, this template will be integrated into the OSF.

Subject:
Applied Science
Material Type:
Reading
Author:
Alexander C. DeHaven
Andrew Hall
Brian Brown
Charles R. Ebersole
Courtney K. Soderberg
David Thomas Mellor
Elliott Kruse
Jerome Olsen
Jessica Kosie
K. D. Valentine
Lorne Campbell
Marjan Bakker
Olmo van den Akker
Pamela Davis-Kean
Rodica I. Damian
Stuart J. Ritchie
Thuy-vy Ngugen
William J. Chopik
Sara J. Weston
Date Added:
08/12/2021
Secondary Data Preregistration
Unrestricted Use
Public Domain
Rating
0.0 stars

Preregistration is the process of specifying project details, such as hypotheses, data collection procedures, and analytical decisions, prior to conducting a study. It is designed to make a clearer distinction between data-driven, exploratory work and a-priori, confirmatory work. Both modes of research are valuable, but are easy to unintentionally conflate. See the Preregistration Revolution for more background and recommendations.

For research that uses existing datasets, there is an increased risk of analysts being biased by preliminary trends in the dataset. However, that risk can be balanced by proper blinding to any summary statistics in the dataset and the use of hold out datasets (where the "training" and "validation" datasets are kept separate from each other). See this page for specific recommendations about "split samples" or "hold out" datasets. Finally, if those procedures are not followed, disclosure of possible biases can inform the researcher and her audience about the proper role any results should have (i.e. the results should be deemed mostly exploratory and ideal for additional confirmation).

This project contains a template for creating your preregistration, designed specifically for research using existing data. In the future, this template will be integrated into the OSF.

Subject:
Life Science
Social Science
Material Type:
Reading
Author:
Alexander C. DeHaven
Andrew Hall
Brian Brown
Charles R. Ebersole
Courtney K. Soderberg
David Thomas Mellor
Elliott Kruse
Jerome Olsen
Jessica Kosie
K.D. Valentine
Lorne Campbell
Marjan Bakker
Olmo van den Akker
Pamela Davis-Kean
Rodica I. Damian
Stuart J Ritchie
Thuy-vy Nguyen
William J. Chopik
Sara J. Weston
Date Added:
08/03/2021
Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results
Unrestricted Use
CC BY
Rating
0.0 stars

Background The widespread reluctance to share published research data is often hypothesized to be due to the authors' fear that reanalysis may expose errors in their work or may produce conclusions that contradict their own. However, these hypotheses have not previously been studied systematically. Methods and Findings We related the reluctance to share research data for reanalysis to 1148 statistically significant results reported in 49 papers published in two major psychology journals. We found the reluctance to share data to be associated with weaker evidence (against the null hypothesis of no effect) and a higher prevalence of apparent errors in the reporting of statistical results. The unwillingness to share data was particularly clear when reporting errors had a bearing on statistical significance. Conclusions Our findings on the basis of psychological papers suggest that statistical results are particularly hard to verify when reanalysis is more likely to lead to contrasting conclusions. This highlights the importance of establishing mandatory data archiving policies.

Subject:
Psychology
Social Science
Material Type:
Reading
Provider:
PLOS ONE
Author:
Dylan Molenaar
Jelte M. Wicherts
Marjan Bakker
Date Added:
08/07/2020