What is the Replication Crisis?

The replication crisis is an ever-present issue in scientific research. This article will look at this subject, providing a brief overview of this complex subject.

Image Credit: Tartila/Shutterstock.com

The Replication Crisis: An Overview

The replication crisis, also known as the reproducibility crisis and the replicability crisis, is a crisis that impacts the methodology of scientific research. Over time, it has been realized by several bodies that the results of many scientific studies are hard or almost impossible to accurately reproduce. The reproducibility of empirical data is essential for the scientific method, so difficulties in reproducing the results of a study or theory undermine its credibility.

Because of this, valuable scientific knowledge may be lost. The peer-review process is an integral part of scientific research, and if the data cannot be reproduced it cannot be validated. The crisis particularly affects the medical and social sciences. Considerable efforts in these fields have been performed into revisiting classic results, confirming their viability, and assessing the reasons for their failure if they are found to be unreliable. There is strong evidence from survey data that the crisis impacts every natural science field.

Growing awareness of the problem led to the term being coined in the early 2010s. The identification of the issue and considerations of causes and remedies have led to the new scientific discipline of metascience which uses empirical research methods to examine empirical research practices.

Both obtaining and analyzing data are involved in empirical research. Because of this, there are two categories of consideration concerning reproducibility. Reproducibility is the validation of the analysis and the interpretation of the data. Repeating an experiment or study to obtain new independent data which verifies the results of the original study is called replication.

Causes

There are a few causes of low reproducibility. A major cause is publication bias, where studies can become statistically skewed in the chase for significant results and overwhelming the correct results. Additional causes include questionable data analysis practices such as researcher degrees of freedom, data dredging, and HARKing. There can also be a failure to adhere to good scientific practice due to the generation of unprecedented rates of new publications and data, and the desire to publish or perish.

Scope

The journal Nature highlighted the scope of the issue in 2016 with a poll of 1,500 scientists. 70% of respondents reported that they had failed to reproduce the results of at least one of their peer’s studies. 87% of chemists, 69% of physicists and engineers, 77% of biologists, 64% of environmental and earth scientists, 67% of medical researchers, and 62% of all other respondents reported this issue. 50% had failed to reproduce one of their own experiments.

Additionally, several respondents reported that academic editors and reviewers told them to tone down comparisons to original studies when they reported failed replications. Sometimes the problem appears to be deliberate, as well – out of respondents to a 2009 study, 2% admitted that they had falsified studies at least once, and 14% said they knew someone who did. According to one study, this misconduct is more prevalent in medical research.

A 2021 study found that papers with reproducible results tend to be cited less than papers with findings that cannot be replicated in leading journals. The authors of the study stated that one of the reasons for this is that the review team may face a trade-off between maintaining standards and how interesting the paper is. Additionally, the authors stated that some papers create hype, and there is pressure to receive grants and court media coverage.

Psychology: A Field at the Center of the Controversy

Psychology has a particular problem with the reproducibility of study data and results. A 2018 meta-analysis study of 200 papers concluded that psychological research, on average, is littered with examples of low statistical robustness. Social psychology has been particularly implicated, but the fields of educational research, clinical psychology, and developmental psychology all have problems with reproducibility.

A study published in Nature Human Behavior failed to replicate 13 out of 21 behavioral and social science papers published in Science and Nature, two leading journals. In another study, 186 researchers from 60 laboratories failed to replicate 14 out of 28 findings despite large sample sizes. One explanation is the prevalence of questionable research practices in the field of psychology, including selective reporting, partial publication of data, post-hoc storytelling, and optional stopping.

In a study, a majority of 2,000 psychologists who responded admitted to using at least one questionable research practice. Solutions are difficult to realize in the field, but large-scale collaboration between researchers to make their data freely accessible is becoming more common practice.

Medical Science

Non-reproducibility of results in the field of medical science can have serious consequences. The US Food and Drug Administration, for example, discovered that 10-20% of medical studies between the years 1977 and 1990 were flawed. In 2012, Begley and Ellis discovered that out of 53 pre-clinical cancer studies, only 11% could be successfully replicated.

A survey by Nature discovered that more than 70% of respondents had failed to reproduce the results of at least one other study. Over half could not replicate at least one of their experiments. John Ioannidis of Stanford University has called for widespread reform, with one solution being to build a more patient-centric model of medicine instead of the current practices which favor the needs of physicians, sponsors, and investigators.

The Future

For research to be valid, the results and data must be reproducible, and currently, there is a crisis in multiple scientific fields. There are no easy solutions to the replication crisis, but researchers, journals, academic editors, patients, and advocacy groups can play a role in ensuring that robust standards are adhered to in scientific research. With increasing focus on the replication crisis, there may be a way forward that will improve the validity and quality of research exponentially.

References:

• Serra-Garcia, M & Gneezy, U (2021) Nonreplicable publications are cited more than replicable ones Sci Adv 7(21) eabd1705 [online] Accessed 27^th Jan 2022 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8139580/
• Stanley, T.D. et al. (2018). What Meta-Analyses Reveal About the Replicability of Psychological Research Psychological Bulletin, 144, 1325–1346 [online] Accessed 27^th Jan 2022. https://pubmed.ncbi.nlm.nih.gov/30321017/
• Camerer, C.F et al. (2018) Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 Nature Human Behavior 2 pp. 637-644 [online] Accessed 27^th Jan 2022 https://www.nature.com/articles/s41562-018-0399-z/
• Ionnaidia, J.P.A (2016) Why Most Clinical Research is Not Useful PLoS Med 1396) e1002049 [online] Accessed 27th Jan 2022 https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002049

Further Reading

• All Research Content

Written by

Reginald Davey

Reg Davey is a freelance copywriter and editor based in Nottingham in the United Kingdom. Writing for News Medical represents the coming together of various interests and fields he has been interested and involved in over the years, including Microbiology, Biomedical Sciences, and Environmental Science.