Statistical significance

Sometimes we look for patterns. We want to know if a result is real. Is it a real change or just luck?

Scientists use math to check this. They use a special rule. This rule is called significance. It helps them decide if a pattern is true.

A common rule is five percent. If a result is very rare, it might be real. It might not be a mistake.

Some people think this rule is too easy. They want to use even stricter rules. This helps them find the truth more clearly.

Math helps us learn about the world. It helps us see what is real.

When scientists study the world, they look for real patterns. They want to know if a change is real. Or is it just a lucky guess?

To find out, they use a math tool. This tool is called statistical significance. It helps them test a null hypothesis. A null hypothesis is the idea that no real effect exists. It is the idea that nothing special is happening.

Scientists use a number called a p-value. This number tells them how rare a result is. They also pick a cutoff level. This level is called the significance level. A common cutoff is 5%. If the p-value is less than 5%, the result is significant. This means the effect might be real for the whole group.

In some fields, the rules are much stricter. Particle physics uses a very high bar. They look for a result that happens only once in 3.5 million tries. This helped them find the Higgs boson particle.

But being significant does not always mean a result is important. A change can be real but very small. Some scientists think we rely on these rules too much. They want to use new ways to find the truth.

Scientists often want to know if a discovery is real. They start by testing a null hypothesis. This is the idea that nothing special is happening. It assumes that any pattern is just a lucky guess. This guess is called a sampling error. To test this, they use a tool called statistical significance.

This tool helps them decide if a result is truly special. It helps them decide whether to reject the null hypothesis.

Researchers use two main numbers to make this choice. The first is the p-value. This number shows how rare a result is. It is the chance of seeing that result if the null hypothesis were true. The second number is the significance level. This is a cutoff chosen before the study begins. Usually, this level is set at 5% or even lower. If the p-value is less than the significance level, the result is significant.

People have worked on these ideas for a long time. In the 18th century, John Arbuthnot and Pierre-Simon Laplace studied birth ratios. They looked at the chance of male and female births. Later, in 1925, Ronald Fisher wrote about tests of significance. He suggested using a cutoff of one in twenty, or 0.05. In 1933, Jerzy Neyman and Egon Pearson added more rules. They named the cutoff the significance level. Neyman also introduced confidence levels and intervals in 1937.

Different types of science use different rules for these tests. Some fields use a one-tailed test to look in one direction. Others use a two-tailed test to look at both ends. In particle physics, the rules are much stricter. They use a very high bar called 5 sigma. This is like a p-value of about 1 in 3.5 million. This strict rule helped scientists find the Higgs boson particle. In other areas, like genome studies, the levels can be very tiny.

It is important to remember that significance is not the same as importance. A result can be statistically significant but have a tiny effect. This is called practical significance. Sometimes, a significant result is just a false positive. This means the result was actually a mistake. Because of this, some scientists want to change the rules. In 2016, the American Statistical Association spoke about these issues. They want to make sure the scientific process stays accurate.

Statistical significance is a mathematical tool used in hypothesis testing. It helps researchers decide if a result is likely real or just a random accident. In any experiment, scientists look at a sample from a larger population. Because samples are small, they might show patterns that do not exist in the whole group. This is known as sampling error. Statistical significance provides a formal way to judge if an observed effect reflects the true population. It is a way to decide whether to reject or retain the null hypothesis. The null hypothesis is the starting assumption that no real effect exists.

To use this tool, researchers rely on two specific values. The first is the p-value. The p-value is the probability of seeing a result at least as extreme as the one observed, assuming the null hypothesis is true. The second value is the significance level, denoted by the Greek letter alpha (α). This is a threshold chosen before any data is collected. If the p-value is less than or equal to alpha, the result is called statistically significant. This means the researcher rejects the null hypothesis. However, alpha also represents the probability of a type I error. A type I error, or a false positive, happens when a researcher rejects the null hypothesis even though it was actually true.

Researchers can set up these tests in different ways depending on their questions. In a two-tailed test, the rejection region is split between both ends of the sampling distribution.

This means the significance level is divided into two equal parts at each tail. A one-tailed test puts the entire rejection region at one end. This is used when a researcher predicts a specific direction, such as whether a group is heavier or a student performs better. A one-tailed test can be more powerful because the rejection region is concentrated. However, if the researcher's predicted direction is wrong, the one-tailed test has no power at all.

The history of these ideas spans several centuries. In the 18th century, John Arbuthnot and Pierre-Simon Laplace worked on these concepts. They calculated p-values for the human sex ratio at birth. They assumed a null hypothesis where the probability of male and female births was equal. In 1925, Ronald Fisher advanced the field with his book, *Statistical Methods for Research Workers*. He introduced "tests of significance" and suggested a cutoff of 0.05. Fisher did not believe this 0.05 value should always be fixed. In 1933, Jerzy Neyman and Egon Pearson expanded these ideas. They named the cutoff the significance level and insisted it be set before data collection. Neyman also introduced confidence levels and confidence intervals in 1937.

Different scientific fields use different standards for significance. In particle physics, the requirements are extremely strict. Scientists often use a threshold called 5 sigma (5σ). This corresponds to a p-value of approximately 1 in 3.5 million. This high standard was used to confirm the existence of the Higgs boson particle. In other fields, like genome-wide association studies, significance levels can be even lower because so many tests are performed. In contrast, social sciences might use more standard thresholds, though some journals have even banned significance testing entirely to encourage deeper analysis.

It is vital to distinguish between statistical significance and practical significance. A result can be statistically significant without being important in the real world. This is often measured by effect size, which quantifies the strength of an effect. For example, a medicine might have a statistically significant effect, but the actual improvement might be too small to matter to a patient. This practical importance is sometimes called clinical significance. Researchers are encouraged to report effect sizes, such as Cohen's d, alongside p-values to provide a complete picture of their findings.

There are many ongoing debates about how to use these tools correctly. Some researchers worry about reproducibility, as many significant results turn out to be false positives. In 2016, the American Statistical Association issued a statement warning that using p-values as a simple license for scientific truth can distort science. Some experts have even proposed changing the standard threshold from 0.05 to 0.005 to improve accuracy. Others suggest moving away from thresholds entirely and treating p-values as continuous indices. These discussions aim to ensure that science remains a reliable way to understand the world.