Replication crisis

The replication crisis is a widespread challenge in the scientific community. It refers to the failure of many published research results to be reproduced by other scientists. Reproducibility is a cornerstone of the scientific method. This means that for a discovery to be considered true, it must be able to be found again through repeated testing. When researchers cannot recreate the same results, it undermines the credibility of scientific theories. This issue challenges large parts of our current scientific knowledge. While it is most visible in psychology and medicine, data shows that other natural and social sciences are also affected.

To understand this crisis, we must distinguish between two different processes: reproducibility and replication. In a narrow sense, reproducibility means reexamining the analysis of an existing set of data to see if the math holds up. Replication is a broader step. It involves repeating an entire experiment or study using new, independent data to verify the original conclusions.

There are also different types of replication. Direct replication repeats the original procedures as closely as possible. Systematic replication involves making intentional changes to the study. Conceptual replication tests a hypothesis using different procedures to see if the idea holds true in different ways.

Science uses statistics to decide if a finding is real or just a coincidence. Scientists often start with a null hypothesis. This is a starting assumption that there is no relationship between two things. For example, a null hypothesis might state that a specific drug does not change a patient's recovery rate. Researchers then calculate a test statistic to see how much the data deviates from this assumption. They look for a p-value, which represents the probability of getting these results by chance alone if the null hypothesis were true.

If the p-value is below a certain threshold, such as 0.05, the result is called statistically significant.

A p-value of 0.05 means there is a 5% chance of a false positive. A false positive occurs when a researcher claims an effect exists when it actually does not. Some fields use stricter thresholds, like 0.01 or 0.001, to reduce this risk. However, using a smaller threshold often requires much larger sample sizes. It also increases the chance of a false negative. A false negative happens when a real effect exists, but the test fails to detect it.

This delicate balance is a major part of how scientific testing works.

The term "replication crisis" emerged in the early 2010s due to several specific failures. One major event involved social psychologist John Bargh. His famous 1996 study on "elderly-walking" could not be reproduced in direct replication attempts. This led to intense disagreements within the field. Another event involved Daryl Bem, who published studies suggesting evidence for extrasensory perception. When his data was reanalyzed, no evidence for the phenomenon remained. Furthermore, scientists at biotech companies like Amgen and Bayer reported very low replication rates in cancer research. They found that only 11% to 20% of landmark findings in preclinical oncology could be reproduced.

Many researchers have also pointed to "p-hacking" as a cause of the crisis. P-hacking occurs when researchers exploit the flexibility in how they collect and report data. By trying different ways of looking at the numbers, they can increase the chance of finding a statistically significant result that isn't actually real.

This practice can lead to a high number of false positives in published literature. Because of these issues, many scholars have declared a "crisis of confidence" in certain scientific fields. This has forced the community to look closely at how research is conducted and reported.

In response to these challenges, a new scientific discipline called metascience has emerged. Metascience uses empirical research methods to study the practice of research itself. It looks at how scientists work, how they report data, and how they use statistics. The goal is to find ways to improve the reliability of all scientific work. By studying the methods used to create knowledge, metascience aims to fix the flaws that lead to the replication crisis. This helps ensure that the scientific method remains a strong and trustworthy way to understand our world.