Item Details

Promoting the Benefits and Clarifying Misconceptions about Preregistration, Preprints, and Open Science for the Cognitive Science of Religion

Issue: (0) ADVANCE ACCESS TO FORTHCOMING ARTICLES

Journal: Journal for the Cognitive Science of Religion

Subject Areas: Religious Studies Cognitive Studies Linguistics

DOI: 10.1558/jcsr.38713

Abstract:

Adopting newly proposed “open science” reforms to improve transparency and increase rigor is hard and can make us, as researchers, feel vulnerable. Nonetheless, these reforms are vital to improving the overall quality and confidence of our collective research. We have, through  our own experiences, found that preregistration and detailed analysisplans can help to identify, and potentially avoid, errors. Prepublication has similarly helped us to collaborate and receive feedback on manuscripts, particularly during prolonged periods of peer-review. The Cognitive Science of Religion (CSR) field is constituted by a diversity of scholars and disciplines, and thus faces somewhat unique challenges in its efforts to establish common practices, standards, and research terminology. In this paper we offer an introduction to the open science reforms of preregistration and prepublication specifically orientated towards the CSR field. We discuss common concerns around these practices, highlighting areas of misunderstanding while conceding and discussing genuine limitations. We conclude with voluntary, low-investment recommendations for best-practices with regards to preregistration and preprints for the field of the Cognitive Science of Religion.

Author: Christopher Kavanagh, Rohan Kapitany

View Full Text

References :

Abdill, Richard J., and Ran Blekhman. 2019. “Tracking the Popularity and Outcomes of All BioRxiv Preprints.” BioRxiv, 515643: 1–21. https://doi.org/10.1101/515643

Allen, Christopher, and David M. A. Mehler. 2019. “Open Science Challenges, Benefits and Tips in Early Career and Beyond.” PLOS Biology 17 (5): e3000246. https://doi.org/10.1371/journal.pbio.3000246

Arjó, Gemma, Manuel Portero, Carme Piñol, Juan Viñas, Xavier Matias-Guiu, Teresa Capell, Andrew Bartholomaeus, Wayne Parrott and Paul Christou. 2013. “Plurality of Opinion, Scientific Discourse and Pseudoscience: An In-Depth Analysis of the Séralini et al. Study Claiming That RoundupTM Ready Corn or the Herbicide RoundupTM Cause Cancer in Rats.” Transgenic Research 22(2): 255–267. https://doi.org/10.1007/s11248-013-9692-9

Atkinson, Quentin D., Andrew J. Latham and Joseph A. Watts. 2015. “Are Big Gods a Big Deal in the Emergence of Big Groups?” Religion, Brain & Behavior 5(4): 266–274.

Baumard, Nicolas, Alexandre Hyafil, Ian Morris and Pascal Boyer. 2014. “Increased Affluence Explains the Emergence of Ascetic Wisdoms and Moralizing Religions.” Current Biology 25(1): 10–15. https://doi.org/10.1016/j.cub.2014.10.063

Begley, C. Glenn, and Lee M. Ellis. 2012. “Drug Development: Raise Standards for Preclinical Cancer Research.” Nature 483(7391): 531–533.

Beheim, Bret, Quentin Atkinson, Joseph Bulbulia, Will Gervais, Russell Gray, Joseph Henrich, Martin Lang et al. 2019. “Corrected Analyses Show That Moralizing Gods Precede Complex Societies but Serious Data Concerns Remain.” Preprint. PsyArXiv, jwa2n: 125. https://doi.org/10.31234/osf.io/jwa2n

Bourne, Philip E., Jessica K. Polka, Ronald D. Vale and Robert Kiley. 2017. “Ten Simple Rules to Consider Regarding Preprint Submission.” PLoS Computational Biology 13(5): e1005473.

Bulbulia, Joseph, Michael L. Spezio, Richard Sosis and Wesley J. Wildman. 2016. “Standards for Publishing in Religion, Brain & Behavior.” Religion, Brain & Behavior 6(4): 275–277. https://doi.org/10.1080/2153599X.2016.1227123

Button, Katherine S., John P. A. Ioannidis, Claire Mokrysz, Brian A. Nosek, Jonathan Flint, Emma S. J. Robinson and Marcus R. Munafo. 2013. “Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience.” Nature Review Neuroscience 14(5): 365–376.

Camerer, Colin F., Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler et al. 2016. “Evaluating Replicability of Laboratory Experiments in Economics.” Science 351(6280): 1433–1436. https://doi.org/10.1126/science.aaf0918

Camerer, Colin F., Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Gideon Nave, Brian A Nosek and Thomas Pfeiffer. 2018. “Evaluating the Replicability of Social Science Experiments in Nature and Science between 2010 and 2015.” Nature Human Behaviour 2(9): 637.

Carlson, Jedidiah, and Kelley Harris. 2020. “Quantifying and Contextualizing the Impact of BioRxiv Preprints through Social Media Audience Segmentation.” BioRxiv, 2020.03.06.981589v1: 145. https://doi.org/10.1101/2020.03.06.981589

Carneiro, Clarissa F. D., Victor G. S. Queiroz, Thiago C. Moulin, Carlos A. M. Carvalho, Clarissa B. Haas, Danielle Rayêe, David E. Henshall, Evandro A. De-Souza, Felippe Espinelli and Flávia Z. Boos. 2019. “Comparing Quality of Reporting between Preprints and Peer-Reviewed Articles in the Biomedical Literature.” BioRxiv, 581892v3: 1–15. https://doi.org/10.1101/581892

Chambers, Christopher D., Eva Feredoes, Suresh Daniel Muthukumaraswamy and Peter Etchells. 2014. “Instead of ‘Playing the Game’ It Is Time to Change the Rules: Registered Reports at AIMS Neuroscience and Beyond.” AIMS Neuroscience 1(1): 4–17.

Charles, Sarah J., James E. Bartlett, Kyle J. Messick, Thomas J. Coleman III and Alex Uzdavines. 2019. “Researcher Degrees of Freedom in the Psychology of Religion.” The International Journal for the Psychology of Religion 29(4): 230–245.

Christensen, Garret, Zenan Wang, Elizabeth Levy Paluck, Nicholas Swanson, David J. Birke, Edward Miguel and Rebecca Littman. 2019. “Open Science Practices Are on the Rise: The State of Social Science (3S) Survey.” Preprint. MetaArXiv, 5rksu: 164. https://doi.org/10.31222/osf.io/5rksu

Claesen, Aline, Sara Lucia Brazuna Tavares Gomes, Francis Tuerlinckx and Wolf Vanpaemel. 2019. “Preregistration: Comparing Dream to Reality.” Preprint. PsyArXiv, d8wex: 124. https://doi.org/10.31234/osf.io/d8wex

Cohen, Jacob. 1994. “The Earth Is Round (p \textless.05).” American Psychologist 49(12): 997–1003.

 

Complexity Science Hub Vienna. 2019. “Complex Societies Gave Birth to Big Gods, Not the Other Way Around.” Science Daily. 20 March. https://www.sciencedaily.com/releases/2019/03/190320141116.htm

Errington, Timothy M, Elizabeth Iorns, William Gunn, Fraser Elisabeth Tan, Joelle Lomax and Brian A Nosek. 2014. “Science Forum: An Open Investigation of the Reproducibility of Cancer Biology Research.” Elife 3: e04333.

Fanelli, Daniele. 2012. “Negative Results Are Disappearing from Most Disciplines and Countries.” Scientometrics 90(3): 891–904.

Fecher, Benedikt, and Sascha Friesike. 2014. “Open Science: One Term, Five Schools of Thought.” In Opening Science, edited by S. Bartling and S. Friesike, 17–47. Basel: Springer.

Ferguson, Christopher J. and Moritz Heene. 2012. “A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science’s Aversion to the Null.” Perspectives on Psychological Science 7(6): 555–561.

Fiedler, Klaus, and Norbert Schwarz. 2016. “Questionable Research Practices Revisited.” Social Psychological and Personality Science 7(1): 45–52.

Fiske, Susan T. 2016. “A Call to Change Science’s Culture of Shaming.” APS Observer 29(9). https://www.psychologicalscience.org/observer/
a-call-to-change-sciences-culture-of-shaming

FOSTER Open Science. 2020. “About FOSTER.” Fosteropenscience.Eu. 2020. https://www.fosteropenscience.eu/about

Gelman, Andrew, and Eric Loken. 2013. “The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No ‘Fishing Expedition’ or ‘p-Hacking’ and the Research Hypothesis Was Posited Ahead of Time.” Department of Statistics, Columbia University.

Gervais, Will. 2017. “Post Publication Peer Review.” WillGervais.Com . 2 March. http://willgervais.com/blog/2017/3/2/post-publication-peer-review

Gervais, Will, and Ara Norenzayan. 2012. “Analytic Thinking Promotes Religious Disbelief.” Science 336(6080): 493–496.

———. 2018. “Analytic Atheism Revisited.” Nature Human Behaviour 2(9): 609. https://doi.org/10.1038/s41562-018-0426-0

Giles, Jim. 2003. “Preprint Server Seeks Way to Halt Plagiarists.” Nature 426(6962): 7. https://doi.org/10.1038/426007a

Ginsparg, Paul. 2011. “ArXiv at 20.” Nature 476(August): 145147.

Goldin-Meadow, Susan. 2016. “Why Preregistration Makes Me Nervous.” APS Observer 29 (7). https://www.psychologicalscience.org/observer/why-preregistration-makes-me-nervous

Gomes, Cristina M., and Michael E. McCullough. 2015. “The Effects of Implicit Religious Primes on Dictator Game Allocations: A Preregistered Replication Experiment.” Journal of Experimental Psychology: General 144(6): e94.

Gregory, Justin P., and Tyler S. Greenway. 2017. “The Mnemonic of Intuitive Ontology Violation Is Not the Distinctiveness Effect: Evidence from a Broad Age Spectrum of Persons in the UK and China during a Free-Recall Task.” Journal of Cognition and Culture 17(1–2): 169–197.

Hagger, Martin S., Nikos L. D. Chatzisarantis, Hugo Alberts, Calvin O. Anggono, Cedric Batailler, Angela R. Birt, Ralf Brand, Mark J. Brandt, Gene Brewer and Sabrina Bruyneel. 2016. “A Multilab Preregistered Replication of the Ego-Depletion Effect.” Perspectives on Psychological Science 11(4): 546–573.

Hoogeveen, S., E. J. Wagenmakers, A. C. Kay and M. van Elk. 2018. “Compensatory Control and Religious Beliefs: A Registered Replication Report across Two Countries.” Comprehensive Results in Social Psychology 3(3): 240–265.

Hoogeveen, Suzanne, Lukas Snoek and M. van Elk. 2020. “Religious Belief and Cognitive Conflict Sensitivity: A Preregistered FMRI Study.” Cortex 129(August): 247–265. https://doi.org/10.1016/j.cortex.2020.04.011

Ioannidis, John P. A. 2005a. “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research.” Journal of the American Medical Association 294(2): 218–228. https://doi.org/10.1001/jama.294.2.218

———. 2005b. “Why Most Published Research Findings Are False.” PLoS Medicine 2(8): e124: 0696-0701. https://doi.org/10.1371/journal.pmed.0020124

Jerome, F. 1989. “Science by Press Conference.” Technology Review 42(5): 72–73.

John, Leslie K., George Loewenstein and Drazen Prelec. 2012. “Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling.” Psychological Science 23(5): 524–532.

Kaplan, Robert M., and Veronica L. Irvin. 2015. “Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time.” PLoS ONE 10(8): e0132382. https://doi.org/10.1371/journal.pone.0132382

Kerr, Norbert L. 1998. “HARKing: Hypothesizing after the Results Are Known.” Personality and Social Psychology Review 2(3): 196–217.

Kidwell, Mallory C., Ljiljana B. Lazarević, Erica Baranski, Tom E. Hardwicke, Sarah Piechowski, Lina-Sophia Falkenberg, Curtis Kennett, Agnieszka Slowik, Carina Sonnleitner and Chelsey Hess-Holden. 2016. “Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency.” PLoS Biology 14(5): e1002456.

Kirkham, Jamie J., Naomi Penfold, Fiona Murphy, Isabelle Boutron, John PA Ioannidis, Jessica K. Polka and David Moher. 2020. “A Systematic Examination of Preprint Platforms for Use in the Medical and Biomedical Sciences Setting.” BioRxiv, April, 2020.04.27.063578. https://doi.org/10.1101/2020.04.27.063578

Klein, Richard A., Michelangelo Vianello, Fred Hasselman, Byron G. Adams, Reginald B. Adams, Sinan Alper, Mark Aveyard et al. 2018. “Many Labs 2: Investigating Variation in Replicability Across Samples and Settings.” Advances in Methods and Practices in Psychological Science 1(4): 443–490. https://doi.org/10.1177/2515245918810225

Koole, Sander L., and Daniël Lakens. 2012. “Rewarding Replications: A Sure and Simple Way to Improve Psychological Science.” Perspectives on Psychological Science 7(6): 608–614. https://doi.org/10.1177/1745691612462586

Kunda, Ziva. 1987. “Motivated Inference: Self-Serving Generation and Evaluation of Causal Theories.” Journal of Personality and Social Psychology 53(4): 636–647. https://doi.org/10.1037/0022-3514.53.4.636

Laakso, Mikael. 2014. “Green Open Access Policies of Scholarly Journal Publishers: A Study of What, When, and Where Self-Archiving Is Allowed.” Scientometrics 99(2): 475–494. https://doi.org/10.1007/s11192-013-1205-3

Lakens, Daniel. 2019. “The Value of Preregistration for Psychological Science: A Conceptual Analysis.” Japanese Psychological Review 62(3): 221–230.

Levy Paluck, Betsy. 2018. “Open Science Practices Are on the Rise across Four Social Science Disciplines.” Presented at the 2018 Berkeley Initiative for Transparency in the Social Sciences, David Brower Center, Berkeley, 30 November. https://osf.io/kvbnh/

Martin, Luther H., and Donald Wiebe. 2017. Religion Explained?: The Cognitive Science of Religion after Twenty-Five Years. London, UK: Bloomsbury.

Matthews, Robert. 2000. “Storks Deliver Babies (P= 0.008).” Teaching Statistics 22(2): 36–38. https://doi.org/10.1111/1467-9639.00013

Meehl, Paul E. 1967. “Theory-Testing in Psychology and Physics: A Methodological Paradox.” Philosophy of Science 34(2): 103–115.

Miyatake, Sanae, and Masataka Higuchi. 2017. “Does Religious Priming Increase the Prosocial Behaviour of a Japanese Sample in an Anonymous Economic Game?” Asian Journal of Social Psychology 20(1): 54–59. https://doi.org/10.1111/ajsp.12164

Moore, Andrew. 2006. “Bad Science in the Headlines: Who Takes Responsibility When Science Is Distorted in the Mass Media?” EMBO Reports 7(12): 1193–1196.

Nickerson, Raymond S. 1998. “Confirmation Bias: A Ubiquitous Phenomenon in Many Guises.” Review of General Psychology 2(2): 175–220.

Norenzayan, Ara. 2013. Big Gods: How Religion Transformed Cooperation and Conflict. Princeton, NJ: Princeton University Press.

Nosek, Brian A., George Alter, George C. Banks, Denny Borsboom, Sara D. Bowman, Steven J. Breckler, Stuart Buck, Christopher D. Chambers, Gilbert Chin and Garret Christensen. 2015. “Promoting an Open Research Culture.” Science 348(6242): 1422–1425.

———. 2016. “Transparency and Openness Promotion (TOP) Guidelines.” Center for Open Sciencehttps://osf.io/9f6gx/

Nosek, Brian A., Charles R. Ebersole, Alexander C. DeHaven and David T. Mellor. 2018. “The Preregistration Revolution.” Proceedings of the National Academy of Sciences 115(11): 26002606. https://doi.org/10.1073/pnas.1708274114

Nosek, Brian A., and Daniël Lakens. 2014. “Registered Reports.” Social Psychology 45(3): 137–141. https://doi.org/10.1027/1864-9335/a000192

Nosek, Brian A., Jeffrey R. Spies and Matt Motyl. 2012. “Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth over Publishability.” Perspectives on Psychological Science 7(6): 615–631.

Open Science Collaboration. 2012. “An Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science.” Perspectives on Psychological Science 7(6): 657–660.

———. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349(6251): aac4716.

Pfungst, Oskar. 1911. Clever Hans:(The Horse of Mr. Von Osten.) a Contribution to Experimental Animal and Human Psychology. New York: Holt, Rinehart and Winston.

Powell, Kendall. 2016. “Does It Take Too Long to Publish Research?” Nature News 530(7589): 148–151.

 

Purzycki, Benjamin Grant, and Aiyana K. Willard. 2015. “Accounting for Variation and Stability in Religious Cognition.” Religion, Brain & Behavior, April, 60–68. https://doi.org/10.1080/2153599X.2015.1009850

Quintana, Daniel S., and Nhat Trung Doan. 2016. “Twitter Article Mentions and Citations: An Exploratory Analysis of Publications in the American Journal of Psychiatry.” American Journal of Psychiatry 173(2): 194. https://doi.org/10.1176/appi.ajp.2015.15101341

Ranehill, Eva, Anna Dreber, Magnus Johannesson, Susanne Leiberg, Sunhae Sul and Roberto A Weber. 2015. “Assessing the Robustness of Power Posing: No Effect on Hormones and Risk Tolerance in a Large Sample of Men and Women.” Psychological Science 26(5): 653–656.

Retraction Watch. 2020. “Retracted Coronavirus (COVID-19) Papers.” Retraction Watchhttps://retractionwatch.com/retracted-coronavirus-
covid-19-papers/

Ritchie, Stuart. 2020. Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth. London: The Bodley Head.

Rödder, Simone, Martina Franzen and Peter Weingart. 2011. The Sciences’ Media Connection–Public Communication and Its Repercussions. Berlin: Springer Science and Business Media.