{"id":1006,"date":"2019-08-13T19:06:41","date_gmt":"2019-08-13T19:06:41","guid":{"rendered":"https:\/\/opentextbc.ca\/researchmethods\/chapter\/from-the-replicability-crisis-to-open-science-practices\/"},"modified":"2019-11-05T18:02:47","modified_gmt":"2019-11-05T18:02:47","slug":"from-the-replicability-crisis-to-open-science-practices","status":"publish","type":"chapter","link":"https:\/\/opentextbc.ca\/researchmethods\/chapter\/from-the-replicability-crisis-to-open-science-practices\/","title":{"raw":"From the \u201cReplicability Crisis\u201d to Open Science Practices","rendered":"From the \u201cReplicability Crisis\u201d to Open Science Practices"},"content":{"raw":"
\r\n

Learning Objectives<\/p>\r\n\r\n<\/header>\r\n

\r\n
    \r\n \t
  1. Describe what is meant by the \"replicability crisis\" in psychology.<\/li>\r\n \t
  2. Describe some questionable\u00a0research practices.<\/li>\r\n \t
  3. Identify\u00a0some ways in which scientific rigour may be increased.<\/li>\r\n \t
  4. Understand the importance of openness in psychological science.<\/li>\r\n<\/ol>\r\n<\/div>\r\n<\/div>\r\nAt the start of this book<\/a> we discussed the \"Many Labs Replication Project\"<\/a>, which failed to replicate the original finding by\u00a0Simone Schnall and her colleagues that washing\u00a0<\/span>one\u2019s hands leads people to view moral transgressions\u00a0as less wrong\u00a0(Schnall, Benton, & Harvey, 2008)[footnote]Schnall, S., Benton, J., & Harvey, S. (2008). With a clean conscience: Cleanliness reduces the severity of moral judgments. Psychological Science, 19<\/em>(12), 1219-1222. doi: 10.1111\/j.1467-9280.2008.02227.x[\/footnote]. Although\u00a0this project is a good illustration<\/span><\/span> of the collaborative and self-correcting nature of science,\u00a0it\u00a0also represents one specific response to psychology's\u00a0recent \u201c[pb_glossary id=\"1222\"]replicability crisis[\/pb_glossary],\u201d\u00a0a phrase that\u00a0refers to the inability of researchers to replicate earlier research findings. Consider for example the results of the Reproducibility Project<\/a>, which involved over 270 psychologists around the world coordinating their efforts to test the reliability of 100 previously published psychological experiments (Aarts et al., 2015)[footnote]Aarts, A. A., Anderson, C. J., Anderson, J., van Assen, M. A. L. M., Attridge, P. R., Attwood, A. S., \u2026 Zuni, K. (2015, September 21). Reproducibility Project: Psychology.<\/em> Retrieved from osf.io\/ezcuj[\/footnote]. Although 97 of the original 100 studies had found statistically significant effects, only 36 of the replications did! Moreover, even the effect sizes of the replications were, on average, half of those found in the original studies (see Figure 13.5). Of course, a failure to replicate a result by itself does not necessarily discredit the original study as differences in the statistical power, populations sampled, and procedures used, or even the effects of moderating variables could\u00a0explain the different results (Yong, 2015)[footnote]Yong, E. (August 27, 2015). How reliable are psychology studies? The Atlantic.<\/em> Retrieved from http:\/\/www.theatlantic.com\/science\/archive\/2015\/08\/psychology-studies-reliability-reproducability-nosek\/402466\/[\/footnote].\r\n\r\n[caption id=\"attachment_190\" align=\"aligncenter\" width=\"490\"]\"Summary<\/a> Figure 13.5 Summary of the Results of the Reproducibility Project [Baker, M. (30 April, 2015). First results from psychology\u2019s largest reproducibility test. Nature News.] [Long Description]<\/a>[\/caption]Although many believe that the failure to replicate research results is an expected characteristic of cumulative scientific progress, others have interpreted this situation as evidence of systematic problems with conventional scholarship in psychology, including a publication bias that favours the discovery and publication of counter-intuitive but statistically significant findings instead of the duller (but incredibly vital) process of replicating previous findings to test their robustness (Aschwanden, 2015[footnote]Aschwanden, C. (2015, August 19). Science isn't broken: It's just a hell of a lot harder than we give it credit for. Fivethirtyeight<\/em>. Retrieved from http:\/\/fivethirtyeight.com\/features\/science-isnt-broken\/[\/footnote]; Frank, 2015[footnote]Frank, M. (2015, August 31). The slower, harder ways to increase reproducibility<\/em>. Retrieved from http:\/\/babieslearninglanguage.blogspot.ie\/2015\/08\/the-slower-harder-ways-to-increase.html[\/footnote]; Pashler & Harris, 2012[footnote]Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments explained. Perspectives on Psychological Science, 7<\/em>(6), 531-536. doi:10.1177\/1745691612463401[\/footnote]; Scherer, 2015[footnote]Scherer, L. (2015, September). Guest post by Laura Scherer.<\/em> Retrieved from http:\/\/sometimesimwrong.typepad.com\/wrong\/2015\/09\/guest-post-by-laura-scherer.html[\/footnote]). Worse still is the suggestion that the low replicability of many studies is evidence of the widespread use of questionable research practices by\u00a0psychological researchers. These may include:\r\n
      \r\n \t
    1. The selective deletion of outliers in order to influence (usually by artificially inflating) statistical relationships among the measured variables.<\/li>\r\n \t
    2. The selective reporting of results, cherry-picking only those findings that support one\u2019s hypotheses.<\/li>\r\n \t
    3. Mining the data without an a priori<\/em> hypothesis, only to claim that a statistically significant result had been originally predicted, a practice referred to as \u201c[pb_glossary id=\"1074\"]HARKing[\/pb_glossary]\u201d or hypothesizing after the results are known (Kerr, 1998[footnote]Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2<\/em>(3), 196-217. doi:10.1207\/s15327957pspr0203_4[\/footnote]).<\/li>\r\n \t
    4. A practice colloquially known as \u201cp<\/em>-hacking<\/a>\u201d (briefly discussed in the previous section), in which a researcher might perform inferential statistical calculations to see if a result was significant before deciding whether to recruit additional participants and collect more data (Head, Holman, Lanfear, Kahn, & Jennions, 2015)[footnote]Head M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p<\/em>-hacking in science. PLoS Biol, 13<\/em>(3): e1002106. doi:10.1371\/journal.pbio.1002106[\/footnote]. As you have learned, the probability of finding a statistically significant result is influenced by the number of participants in the study.<\/li>\r\n \t
    5. Outright fabrication of data (as in the case of Diederik Stapel, described at the start of Chapter 3<\/a>), although this would be\u00a0a case of fraud rather than a \"research practice.\"<\/li>\r\n<\/ol>\r\nIt is important to shed light on these questionable research\u00a0practices to ensure that current and future researchers (such as yourself) understand the damage they wreak to the integrity and reputation of our discipline (see,\u00a0for example,\u00a0the \"Replication Index<\/a>,\" a statistical \"doping test\"\u00a0developed by\u00a0Ulrich Schimmack in 2014 for estimating the replicability of studies, journals, and even specific researchers).\u00a0However, in addition to highlighting what not to do<\/em>, this so-called \u201ccrisis\u201d has also highlighted the importance of enhancing scientific rigour by:\r\n
        \r\n \t
      1. Designing and conducting studies that have sufficient statistical power, in order to increase the reliability of findings.<\/li>\r\n \t
      2. Publishing both null and significant findings (thereby counteracting the publication bias and reducing the file drawer problem).<\/li>\r\n \t
      3. Describing one\u2019s research designs in sufficient detail to enable other researchers to replicate your study using an identical or at least very similar procedure.<\/li>\r\n \t
      4. Conducting high-quality replications and publishing these results (Brandt et al., 2014)[footnote]Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., \u2026 van\u2019t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50,<\/em> 217-224. doi:10.1016\/j.jesp.2013.10.005[\/footnote].<\/li>\r\n<\/ol>\r\nOne particularly promising response to the replicability crisis has been the emergence of [pb_glossary id=\"1061\"]open science practices[\/pb_glossary] that increase the transparency and openness of the scientific enterprise. For example, Psychological Science<\/em> (the flagship journal of the Association for Psychological Science<\/a>) and other journals now issue digital badges to researchers who pre-registered their hypotheses and data analysis plans, openly shared their research materials with other researchers (e.g., to enable attempts at replication), or made available their raw data with other researchers (see Figure 13.6).\r\n\r\n[caption id=\"attachment_1004\" align=\"aligncenter\" width=\"600\"]\"Badges<\/a> Figure 13.6 Digital Badges from the Center for Open Science[\/caption]\r\n\r\nThese initiatives, which have been spearheaded by the Center for Open Science<\/a>, have led to the development of \u201cTransparency and Openness Promotion guidelines\u201d (see Table 13.7) that have since been formally adopted by more than 500 journals and 50 organizations, a list that grows each week. When you add to this the requirements recently\u00a0imposed by federal funding agencies in Canada (the Tri-Council) and the United States (National Science Foundation) concerning the publication of publicly-funded research in open access journals, it certainly appears that the future of science and psychology will be one that embraces greater \u201copenness\u201d (Nosek et al., 2015)[footnote]Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., \u2026 Yarkoni, T. (2015). Promoting an open research culture. Science, 348<\/em>(6242), 1422-1425. doi: 10.1126\/science.aab2374[\/footnote].\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
        Table 13.7 Transparency and Openness Promotion (TOP) Guidelines<\/caption>\r\n
        Criteria<\/th>\r\nLevel 0<\/th>\r\nLevel 1<\/th>\r\nLevel 2<\/th>\r\nLevel 3<\/th>\r\n<\/tr>\r\n
        Citation Standards<\/th>\r\nJournal encourages citation of data, code, and materials, or says nothing<\/td>\r\nJournal describes citation of data in guidelines to authors with clear rules and examples.<\/td>\r\nArticle provides appropriate citation for data and materials used consistent with journal's author guidelines.<\/td>\r\nArticle is not published until providing appropriate citation for data and materials following journal's author guidelines.<\/td>\r\n<\/tr>\r\n
        Data Transparency<\/th>\r\nJournal encourages data sharing, or says nothing<\/td>\r\nArticle states whether data are available, and, if so, where to access them.<\/td>\r\nData must be posted to a trusted repository. Exceptions must be identified at article submission.<\/td>\r\nData must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication.<\/td>\r\n<\/tr>\r\n
        Analytic Methods (Code) Transparency<\/th>\r\nJournal encourages code sharing, or says nothing<\/td>\r\nArticle states whether code is available, and, if so, where to access them.<\/td>\r\nCode must be posted to a trusted repository. Exceptions must be identified at article submission.<\/td>\r\nCode must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication.<\/td>\r\n<\/tr>\r\n
        Research Materials Transparency<\/th>\r\nJournal encourages materials sharing, or says nothing<\/td>\r\nArticle states whether materials are available, and, if so, where to access them.<\/td>\r\nMaterials must be posted to a trusted repository. Exceptions must be identified at article submission.<\/td>\r\nMaterials must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication.<\/td>\r\n<\/tr>\r\n
        Design and Analysis Transparency<\/th>\r\nJournal encourages design and analysis transparency, or says nothing<\/td>\r\nJournal articulates design transparency standards<\/td>\r\nJournal requires adherence to design transparency standards for review and publication<\/td>\r\nJournal requires and enforces adherence to design transparency standards for review and publication<\/td>\r\n<\/tr>\r\n
        Preregistration of studies<\/th>\r\nJournal says nothing<\/td>\r\nJournal encourages preregistration of studies and provides link in article to preregistration if it exists<\/td>\r\nJournal encourages preregistration of studies and provides link in article and certification of meeting preregistration badge requirements<\/td>\r\nJournal requires preregistration of studies and provides link and badge in article to meeting requirements.<\/td>\r\n<\/tr>\r\n
        Preregistration of analysis plans<\/th>\r\nJournal says nothing<\/td>\r\nJournal encourages preanalysis plans and provides link in article to registered analysis plan if it exists<\/td>\r\nJournal encourages preanalysis plans and provides link in article and certification of meeting registered analysis plan badge requirements<\/td>\r\nJournal requires preregistration of studies with analysis plans and provides link and badge in article to meeting requirements.<\/td>\r\n<\/tr>\r\n
        Replication<\/th>\r\nJournal discourages submission of replication studies, or says nothing<\/td>\r\nJournal encourages submission of replication studies<\/td>\r\nJournal encourages submission of replication studies and conducts results blind review<\/td>\r\nJournal uses Registered Reports as a submission option for replication studies with peer review prior to observing the study outcomes.<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n
        \r\n

        Key Takeaways<\/p>\r\n\r\n<\/header>\r\n

        \r\n