Linear regression

Math helps us see how things change together.

Imagine one thing changes, and it makes another thing change too. We can use math to find this link. This is called linear regression. It can use one idea or many ideas to guess. We call it simple if there is only one idea. We use it to make smart guesses. It helps us find patterns in a lot of data. This is a great tool for learning about our world.

Imagine you want to know how one thing affects another.

Perhaps you want to see how time affects the height of a ball. We call this link a relationship. Linear regression is a math tool used to find this link. It looks at data to see how variables move together.

One variable is the response. This is the thing we want to predict. The other variables are the predictors. These are the things that help us make the guess. If we use only one predictor, it is called simple linear regression. If we use many predictors, it is called multiple linear regression.

This tool helps us in two main ways. First, it helps us make smart guesses. We can use a model to predict new data. Second, it helps us explain why things change. It shows us how strong a link is. It can even show if a predictor does not help at all. Sometimes, data has "noise." This is just random error that makes the pattern messy. We use math to find the best line through the noise. This helps us find the true signal in the data.

Imagine you want to see how one thing changes when another thing changes.

You might wonder how the height of a ball changes as it flies through the air. Linear regression is a mathematical way to find this link. It uses data to estimate the relationship between two types of variables. One is called the response variable, which is the thing you want to predict. The other is the explanatory variable, which helps you make that prediction.

This tool helps us understand how much one thing influences another.

To make this work, we look for a pattern in the data points. We assume there is a true relationship hidden under some random noise. This noise is called the error term, or sometimes just noise. It represents all the tiny, random things that affect the result but are not part of our main pattern.

When we use only one predictor, we call it simple linear regression. If we use many predictors at once, it is called multiple linear regression. The goal is to find the best mathematical function to map these points. This function helps us turn messy data into a clear signal.

In the past, linear regression was the first type of regression to be studied very carefully. Scientists used it a lot in practical work. This is because these models are easier to fit than other kinds. It is also easier to figure out the math properties of the results. These models are linear in their unknown parameters. This means the parameters themselves are not raised to a high power. This makes the math much simpler for researchers to solve.

There are many ways to find the best fit for the data. One common way is called the least squares approach. This method tries to make the total error as small as possible. Sometimes, data has large outliers, which are points that do not fit the pattern. In those cases, other methods like ridge regression or lasso might be used. These methods help manage the errors differently. Some models even focus on the median instead of the average. Choosing the right way to fit the model depends on the data you have.

We use these tools to solve real-world problems every day. One main goal is to reduce error when we try to forecast the future. If we have a good model, we can predict new values we have not seen yet. Another goal is to explain why things change. We can see if a specific predictor is actually helpful or just redundant. We can even see if a predictor has no relationship with the response at all. This helps us understand the true strength of the links in our world.

Linear regression is a statistical model used to estimate relationships between variables. It specifically looks at the connection between a scalar response variable and one or more explanatory variables. The response variable is often called the dependent variable, the target, or the regressand. The explanatory variables are known as regressors, predictors, or independent variables.

This model is essential for understanding how changes in one factor might influence another. It is a fundamental tool in both statistics and machine learning. As a supervised machine learning algorithm, it learns from labeled datasets to map data points to optimized functions. These functions allow for predictions on new, unseen datasets.

To understand the mechanism, imagine a relationship hidden beneath layers of random noise. The model assumes that observations result from a true underlying relationship plus random deviations. These deviations are called the error term, disturbance term, or noise.

The noise captures all other factors influencing the response variable that are not included in the predictors. In mathematical terms, the model is expressed as a function of unknown parameters. These parameters are the regression coefficients, often denoted as beta. A key characteristic is that linear regression is linear in its unknown parameters. This means the coefficients appear only to the first power, even if the variables themselves are not linear.

Linear regression models are categorized by the number of explanatory variables they use. A model with exactly one explanatory variable is called simple linear regression. When a model includes two or more explanatory variables, it is known as multiple linear regression. It is important to distinguish this from multivariate linear regression. Multivariate regression predicts multiple correlated dependent variables at once. In contrast, standard linear regression always focuses on a single dependent variable.

Researchers often include a constant in the model, known as the intercept. This intercept is the value of the response variable when all predictors are zero. Even if theory suggests the intercept should be zero, it is often included to satisfy statistical inference requirements.

Historically, linear regression was the first type of regression analysis to be studied rigorously. It has been used extensively in practical applications for a long time. This early adoption happened because these models are easier to fit than non-linear models. Additionally, the statistical properties of the resulting estimators are easier to determine. Scientists found that models depending linearly on unknown parameters provided a reliable foundation for analysis. This ease of use allowed researchers to move quickly from theoretical math to real-world problem solving.

There are several ways to fit these models to a dataset. The most common method is the least squares approach. This method works by minimizing the sum of squared errors to find the best fit. However, the least squares approach can be sensitive to large outliers. If a dataset has many large outliers, the Mean Squared Error (MSE) might cause the model to fit the outliers too closely. In such cases, researchers use robust cost functions. Other methods include least absolute deviations regression or penalized versions like ridge regression and lasso. Ridge regression uses an L2-norm penalty, while lasso uses an L1-norm penalty.

Linear regression serves two primary practical purposes. The first goal is to reduce error in prediction or forecasting. By fitting a model to observed data, we can predict the response for new explanatory values. The second goal is to explain variation in the response variable. This helps quantify the strength of the relationship between variables. It can determine if an explanatory variable has no linear relationship with the response. It can also identify redundant information where one variable provides no new insight beyond what another already provides. This allows scientists to isolate the unique effect of a single predictor.

While powerful, one must be careful when interpreting results. A major pitfall is relying solely on fitted models without looking at the data visually. The Anscombe's quartet is a famous example of this. It consists of four datasets that have nearly identical regression lines and statistical properties. However, when graphed, the datasets look completely different. This demonstrates why visual inspection is necessary alongside mathematical calculation. Understanding the distinction between marginal effects and unique effects is also vital for accurate scientific conclusions.