Regression analysis

Regression analysis is a powerful statistical method used to estimate relationships between variables. It helps researchers understand how a dependent variable changes when one or more independent variables change. The dependent variable is often called the outcome, response variable, or label. The independent variables are known as regressors, predictors, covariates, or features. By finding a mathematical function that best fits the observed data, scientists can model complex real-world processes.

To perform a regression, researchers must first choose a specific mathematical model. This model usually describes the dependent variable as a function of the independent variables plus an error term. The error term, or residual, represents random noise or other factors not included in the model. One of the most common methods is ordinary least squares. This technique calculates a unique line or hyperplane that minimizes the sum of the squared differences between the actual data points and the fitted line.

This process allows researchers to estimate the conditional expectation, which is the population average value for a given set of inputs.

There are several different types of regression depending on the goal of the study. Linear regression is the most frequent form, where the relationship is modeled as a straight line or a linear combination of variables. Other methods include quantile regression, which estimates different location parameters. Nonparametric regression is used when the relationship is non-linear and does not follow a specific shape. Researchers might also use robust regression to handle outliers or errors-in-variables models when predictors contain measurement errors.

The history of regression is filled with famous mathematicians and astronomers. Isaac Newton used an embryonic form of linear regression in 1700 while studying equinoxes. He performed data averaging and forced his regression line to pass through the average point by summing residuals to zero. Later, Adrien-Marie Legendre published the method of least squares in 1805. Carl Friedrich Gauss also published the method in 1809 to determine the orbits of comets and planets.

Gauss later expanded this theory in 1821 with the Gauss-Markov theorem.

The term "regression" actually comes from the field of biology. In the 19th century, Francis Galton studied the heights of human descendants. He observed that the children of very tall ancestors tended to move back toward the normal average. This biological phenomenon is known as regression toward the mean. Later, mathematicians Udny Yule and Karl Pearson extended this concept into a general statistical context. They assumed the variables followed a Gaussian distribution, a concept later refined by R.A. Fisher in the 1920s.

Regression analysis serves two primary purposes: prediction and causal inference. In prediction and forecasting, the method is used to guess future outcomes, which overlaps heavily with machine learning. In causal inference, researchers use it to try to understand if one variable actually causes a change in another. However, regression alone only shows relationships within a fixed dataset. To claim causality, a researcher must carefully justify why the relationship would hold in a new context.

In the mid-20th century, performing these calculations was a massive undertaking. During the 1950s and 1960s, economists used electromechanical desk calculators for their work. Before 1970, it could take up to 24 hours to receive the result from just one regression. Today, the process is nearly instant thanks to modern computers and handheld scientific calculators. Modern research now applies regression to complex data objects like curves, images, and graphs.