Linear regression

Linear regression is a statistical model used to estimate relationships between variables. It specifically looks at the connection between a scalar response variable and one or more explanatory variables. The response variable is often called the dependent variable, the target, or the regressand. The explanatory variables are known as regressors, predictors, or independent variables.

This model is essential for understanding how changes in one factor might influence another. It is a fundamental tool in both statistics and machine learning. As a supervised machine learning algorithm, it learns from labeled datasets to map data points to optimized functions. These functions allow for predictions on new, unseen datasets.

To understand the mechanism, imagine a relationship hidden beneath layers of random noise. The model assumes that observations result from a true underlying relationship plus random deviations. These deviations are called the error term, disturbance term, or noise.

The noise captures all other factors influencing the response variable that are not included in the predictors. In mathematical terms, the model is expressed as a function of unknown parameters. These parameters are the regression coefficients, often denoted as beta. A key characteristic is that linear regression is linear in its unknown parameters. This means the coefficients appear only to the first power, even if the variables themselves are not linear.

Linear regression models are categorized by the number of explanatory variables they use. A model with exactly one explanatory variable is called simple linear regression. When a model includes two or more explanatory variables, it is known as multiple linear regression. It is important to distinguish this from multivariate linear regression. Multivariate regression predicts multiple correlated dependent variables at once. In contrast, standard linear regression always focuses on a single dependent variable.

Researchers often include a constant in the model, known as the intercept. This intercept is the value of the response variable when all predictors are zero. Even if theory suggests the intercept should be zero, it is often included to satisfy statistical inference requirements.

Historically, linear regression was the first type of regression analysis to be studied rigorously. It has been used extensively in practical applications for a long time. This early adoption happened because these models are easier to fit than non-linear models. Additionally, the statistical properties of the resulting estimators are easier to determine.

Scientists found that models depending linearly on unknown parameters provided a reliable foundation for analysis. This ease of use allowed researchers to move quickly from theoretical math to real-world problem solving.

There are several ways to fit these models to a dataset. The most common method is the least squares approach. This method works by minimizing the sum of squared errors to find the best fit.

However, the least squares approach can be sensitive to large outliers. If a dataset has many large outliers, the Mean Squared Error (MSE) might cause the model to fit the outliers too closely. In such cases, researchers use robust cost functions. Other methods include least absolute deviations regression or penalized versions like ridge regression and lasso. Ridge regression uses an L2-norm penalty, while lasso uses an L1-norm penalty.

Linear regression serves two primary practical purposes. The first goal is to reduce error in prediction or forecasting. By fitting a model to observed data, we can predict the response for new explanatory values. The second goal is to explain variation in the response variable. This helps quantify the strength of the relationship between variables.

It can determine if an explanatory variable has no linear relationship with the response. It can also identify redundant information where one variable provides no new insight beyond what another already provides. This allows scientists to isolate the unique effect of a single predictor.

While powerful, one must be careful when interpreting results. A major pitfall is relying solely on fitted models without looking at the data visually.

The Anscombe's quartet is a famous example of this. It consists of four datasets that have nearly identical regression lines and statistical properties. However, when graphed, the datasets look completely different. This demonstrates why visual inspection is necessary alongside mathematical calculation. Understanding the distinction between marginal effects and unique effects is also vital for accurate scientific conclusions.