Replication crisis

The replication crisis, also referred to as the reproducibility crisis and the replicability crisis, is a problem in science revolving around researchers’ inability to replicate the results of a previous experiment. Replication of results in science is an important aspect of scientific validity and an essential part of the scientific method. Without replication, the conclusions of scientific studies are not dependable. The replication crisis emerged in the early 2010s following the revelation that the findings of a number of highly regarded social science experiments could not be reproduced by other researchers replicating the scientific methods utilized in the original experiment. Although the replication crisis affects many areas of science, it particularly affects the social sciences, including psychology. Experts have suggested a number of ways to remedy the replication crisis, such as registering reports and making data openly available. Others deny the existence of the replication crisis, blaming distorted views and sensationalist media.

Background

Replicability is a crucial element of the scientific method, a system of methodical procedures designed to solve a specific problem logically. The scientific method is used by researchers to understand a variety of problems and by scientists from a range of scientific fields. The scientific method consists of five basic steps:

• The scientist makes an observation.
• The scientist asks a question about this observation.
• The scientist forms a hypothesis, or indefinite explanation that can be tested, about the observation.
• The scientist makes a prediction based on this hypothesis.
• The scientist tests the prediction by collecting data through experimentation.

When the results of an experiment successfully support a hypothesis, it becomes a theory. This means the hypothesis is very likely correct, but the nature of science does not allow for absolutes, so it is possible a scientific theory will not always hold up completely. Not every experiment utilizes every aspect of the scientific method, and sometimes the steps are performed out of order, but scientists align their experiments to the principles of the scientific method for the sake of consistency. Since its establishment, the scientific method has informed every scientific theory of repute. Science is considered a cyclical, ever-changing process, however. Just because a hypothesis is supported at one point in time does not make it foolproof. The scientific method is used by researchers to continually improve scientific knowledge. An observation that turns into an experiment often gives rise to another observation in need of experimentation.

Another important element of the scientific method is the concept of reproducibility. An experiment should be able to be replicated by other researchers and produce the same outcome as the original study. If an experiment cannot be replicated, its results must be called into question. Failure to replicate also devalues the entire scientific process.

In the early 2010s, a group of scientists began to notice a pattern of non-reproducibility in many high-profile scientific studies. In experiments ranging from cancer medicine studies to psychological surveys, researchers found that many scientific claims did not hold up when replicated by independent researchers. Some of these studies were highly regarded and appeared in textbooks around the world. As a result, a replication crisis developed within the scientific community, particularly within the fields of medicine and psychology.

Overview

Although the replication crisis affected multiple fields of science, it seemed to primarily affect the psychological sciences. This problem emerged around 2010 following the publication of a study that purportedly showed people were capable of perceiving the future, something that is impossible to do. The researchers who conducted the study utilized the accepted principles of the scientific method, which prompted a number of scientists to call into question the very basis of scientific inquiry. Common practices such as collecting data from a small group of sample subjects was found to be inadequate for observing the true effects of a study, and results thought to be statistically significant based on this small sample were actually statistical flukes.

In light of this reckoning, a number of scientists began to wonder how this issue affects long-established scientific conclusions. In 2015, an article published in Science magazine revealed a major replication issue with dozens of psychological studies published in respected scientific journals. A group of psychologists attempted to replicate one hundred published psychology studies and found that two-thirds of the studies could not be reproduced in line with the original studies’ findings. Only 40 percent of the studies replicated held up, while the remainder were either inconclusive or failed to reproduce altogether. This replication crisis affected some of psychology’s most accepted theories, such as the ego depletion theory regarding the limits of self-control and the Stanford University marshmallow experiment that showed patience was key to success. Such studies have become cornerstones of many other scientific theories and clinical practices, but a lack of reproducibility significantly undermines these findings.

Experts believed the replication crisis was more prominent within the psychological sciences because questionable research practices were more common in the field. Questionable research practices can range from outright fraudulent findings to poor-quality experiments to conflicts of interest. While some of these practices may not be fraudulent in intent, they can provide scientists with more leeway in regard to data collection, interpretation, and reporting. Such practices can lead to data favoritism or false-positive conclusions, with scientists cherry-picking results that are more supportive of their desired outcome. Authors may also be prone to confirmation bias, or the tendency to interpret findings so they confirm the author’s own views. Another issue that potentially affects a study’s reproducibility is inaccessible data. Some scientists do not allow public access to a study’s underlying data, making replication more difficult.

A number of methods have been suggested to help remedy the replication crisis in the years following its emergence. One method supported by much of the scientific community is the practice of pre-registering studies. This would require researchers to submit a description of the studies’ methods to a journal before data collection begins. Reviewers would check to ensure the methods are acceptable and tentatively agree to publish the study as long as the researchers adhere to the approved methods. This would ensure a study is not simply published because of its findings. Experts also suggest larger sample sizes, open access to research data, and the emphasis of replication studies in scientific experimentation.

Bibliography

Fanelli, Danielle. “How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data.” PLOS ONE, vol. 4, no. 5, 2009.

Gelman, Andrew. “Why Does the Replication Crisis Seem Worse in Psychology?” Slate, 3 Oct. 2016, slate.com/technology/2016/10/why-the-replication-crisis-seems-worse-in-psychology.html. Accessed 19 Nov. 2018.

John, Leslie K., et al. “Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling.” Psychological Science, vol. 23, no. 5, 2012, pp. 524–32.

Korbmacher, Max, et al. "The Replication Crisis Has Led to Positive Structural, Procedural, and Community Changes." Communications Psychology, vol. 1, no. 3, 25 July 2023, https://doi.org/10.1038/s44271-023-00003-2. Accessed 6 Nov. 2024.

Palmer, Katie M. “Psychology Is in Crisis over Whether It’s in Crisis.” Wired, 3 Mar. 2016, www.wired.com/2016/03/psychology-crisis-whether-crisis/. Accessed 19 Nov. 2018.

Resnick, Brian. “More Social Science Studies Just Failed to Replicate. Here’s Why This Is Good.” Vox, 27 Aug. 2018, www.vox.com/science-and-health/2018/8/27/17761466/psychology-replication-crisis-nature-social-science. Accessed 19 Nov. 2018.

“The Scientific Method.” Khan Academy, www.khanacademy.org/science/high-school-biology/hs-biology-foundations/hs-biology-and-the-scientific-method/a/the-science-of-biology. Accessed 19 Nov. 2018.

Shapiro, Ari. “When Great Minds Think Unalike: Inside Science’s ‘Replication Crisis.’” NPR, 24 May 2016, www.npr.org/templates/transcript/transcript.php?storyId=477921050. Accessed 19 Nov. 2018.

Vasishth, Shravan. “The Replication Crisis in Science.” The Wire, 29 Dec. 2017, thewire.in/science/replication-crisis-science. Accessed 19 Nov. 2018.

Yong, Ed. “Psychology’s Replication Crisis Can’t Be Wished Away.” The Atlantic, 4 Mar. 2016, www.theatlantic.com/science/archive/2016/03/psychologys-replication-crisis-cant-be-wished-away/472272/. Accessed 19 Nov. 2018.