Statistical inference

Statistical inference is the scientific process of using data analysis to learn about an underlying probability distribution. In many studies, researchers cannot observe every single member of a group. This large group is called the population. Instead, they collect a smaller group called a sample. Statistical inference allows scientists to make educated propositions about the entire population based only on that sample. This is different from descriptive statistics, which only describes the specific data that was actually collected.

The process of inference typically follows a specific sequence of logical steps. First, a researcher must select a statistical model to describe the process that generated the data. This model acts as a mathematical framework for understanding how the sample relates to the population. Second, the researcher uses that model to deduce propositions. These propositions are the conclusions drawn from the data. Common forms of these conclusions include a point estimate, which is a single value that best approximates a population parameter. Another common form is an interval estimate, such as a confidence interval. A 95% confidence interval is a range of values constructed from sample data. If the procedure were repeated many times, 95% of those resulting intervals would contain the true population parameter.

Statisticians categorize statistical models into three distinct levels based on their assumptions. Fully parametric models are the most specific. They assume the data-generation process is described by a family of distributions with a finite number of unknown parameters. An example is assuming a population follows a Normal distribution with an unknown mean and variance. Non-parametric models make much fewer assumptions about the data. For instance, one might estimate a median without assuming a specific shape for the distribution. Semi-parametric models sit between these two approaches. They might use a parametric assumption for one part of the data and a non-parametric approach for another. The well-known Cox model is a famous example of a semi-parametric model.

History shows that the transition from a real-world problem to a mathematical model is a critical step. Sir David Cox noted that translating a subject-matter problem into a statistical model is often the most vital part of any analysis. Many problems in this field are essentially problems of statistical modeling. In the 1950s, the work of Andrey Kolmogorov helped advance the field through approximation theory. This allowed researchers to quantify the error when using one distribution to approximate another. Today, advanced statistics uses functional analysis to study the geometry of probability distributions. This helps scientists measure the distance between different mathematical models.

Accuracy in statistical inference depends heavily on the validity of the initial assumptions. If a researcher uses an incorrect model, the resulting conclusions may be faulty. For example, assuming a population follows a Normal distribution can invalidate certain types of regression-based inference. This is especially dangerous in economic populations, where a Normal distribution might be an unrealistic assumption. To avoid these errors, many experts rely on the central limit theorem. This theorem states that the distribution of a sample mean will be approximately normal if the sample size is very large. This allows for more reliable inferences even when the underlying population is not perfectly normal.

Because it is difficult to know the exact distribution of a sample, mathematicians use various approximation methods. For example, the Berry–Esseen theorem shows that with 10,000 independent samples, a normal distribution can approximate a sample mean to two digits of accuracy. In many practical cases, even 10 independent samples can provide a good approximation. Researchers also use asymptotic theory to study what happens as a sample size tends toward infinity. While limit theorems describe infinite samples, they are often used to guide work on finite samples. To check the magnitude of error in these approximations, scientists often use computer simulations.

Statistical inference also connects deeply to the concept of randomization. In randomized experiments, the data is produced by a specific design that can be mathematically evaluated. In frequentist inference, this allows conclusions to be based on the randomization itself rather than a subjective model. This is very important in survey sampling and the design of experiments. Bayesian inference also utilizes randomization to ensure that samples are exchangeable with the population. By using these rigorous methods, statisticians can turn small, finite sets of observations into powerful tools for understanding the world.