Loss function

A loss function is a mathematical tool used to measure error. In optimization and decision theory, it maps an event or specific variables onto a real number. This number represents a "cost" associated with that event. When mathematicians solve optimization problems, they seek to minimize this loss function. An objective function is a broader term for these tools. It can be a loss function, or it can be its opposite. In some fields, the opposite is called a reward function, a profit function, or a utility function. While a loss function is minimized, these opposite functions are maximized.

Different fields use loss functions in unique ways. In statistics, they are often used for parameter estimation. Here, the loss is a function of the difference between estimated values and true values. In economics, the function might represent economic cost or regret. In classification tasks, it represents the penalty for an incorrect classification. Actuarial science uses it to model insurance benefits paid over premiums. In optimal control, the loss is the penalty for failing to reach a desired value. Financial risk management even maps these functions directly to monetary loss.

One of the most common tools is the quadratic loss function. This is also known as squared error loss, or SEL. It is often used in least squares techniques. This function is mathematically tractable because of the properties of variances. It is also symmetric. This means an error above a target causes the same loss as an error of the same magnitude below the target. If the target is "t," the loss is calculated using a constant "C." For decision-making, the value of this constant can be ignored by setting it to 1. Quadratic loss is the basis for linear regression theory and many common statistics, such as t-tests.

However, quadratic loss has a specific weakness regarding outliers. An outlier is a data point that is very different from the rest of the set. Because the quadratic loss squares the error, it assigns much more importance to outliers than to the true data. When a dataset contains many large outliers, mathematicians use alternatives. These include the Huber loss, the log-cosh loss, and the symmetric mean absolute error, or SMAE. Another simple option is the 0-1 loss function. In information theory, this specific function is known as Hamming distortion.

History shows how these ideas evolved through different thinkers. The concept is as old as the mathematician Laplace. In the middle of the 20th century, Abraham Wald reintroduced the concept to statistics. Leonard J. Savage also contributed by arguing for the idea of regret. Regret is the difference between the decision actually taken and the best decision that could have been made if the circumstances were known in advance. Later, Andranik Tangian showed that useful objective functions, like quadratic and additive ones, are determined by a few indifference points. He used these to distribute budgets for 16 Westfalian universities and subsidies for 271 German regions.

In statistical theory, there are two main ways to look at expected loss. Frequentist theory looks at the expected value with respect to a probability distribution of observed data. This is often called the risk function. Bayesian theory calculates expectation using a prior distribution of the parameter. The goal in the Bayesian approach is to minimize the Bayes Risk. This involves finding an optimal decision, known as the Bayes Rule. The Bayes Rule is useful because it minimizes the average loss over all possible states of nature. This approach is often more efficient than choosing a frequentist optimal decision rule.

Finally, experts like W. Edwards Deming and Nassim Nicholas Taleb suggest looking at empirical reality. They argue that real-world losses are often not "mathematically nice." Real losses might not be continuous, symmetric, or differentiable. For example, arriving slightly early for a plane has little cost. However, arriving slightly late can have a massive cost because you miss the flight. This is an asymmetric and discontinuous situation. In drug dosing, the cost of too little medicine is a lack of efficacy. The cost of too much might be toxicity. Understanding these real-world complexities helps us choose the right mathematical tools.