Estimator

In the field of statistics, researchers often seek to discover a hidden value called a parameter. Because they cannot observe every single part of a population, they must use observed data to make an educated guess. This process requires a formal rule known as an estimator. It is important to distinguish between three distinct concepts: the estimator, the estimand, and the estimate. The estimand is the specific quantity of interest that we want to know. The estimator is the mathematical rule or method used to process data. Finally, the estimate is the actual result produced when that rule is applied to a specific set of data.

Estimators function as mathematical mappings from a sample space to a set of possible estimates. They can be categorized into two primary types: point estimators and interval estimators. A point estimator produces a single-valued result, such as a single number, a vector, or a function. In contrast, an interval estimator provides a range of plausible values rather than one specific point.

When we discuss density estimation, we are often looking for the probability density functions of random variables. In these complex cases, the estimates may be functions that act as point estimates within an infinite-dimensional space.

Estimation theory focuses on the properties of these rules to determine which are most effective. One way to judge an estimator is by its bias. An estimator is considered unbiased if the expected value of its estimates equals the true parameter. If the rule has a systematic tendency to produce results that are too high or too low, it is a biased estimator.

There is also a concept called median-unbiasedness. In this case, the median of the distribution of estimates matches the true value. This means that, over a long period, half of the estimates will be higher than the true value and half will be lower.

Another critical property is variance, which measures how spread out the estimates are. Variance describes how far, on average, the collection of estimates sits from their own expected value. To understand the relationship between these concepts, imagine a person shooting arrows at a target. The bull's-eye represents the true parameter, while the process of shooting represents the estimator. Each individual arrow that hits the target is an estimate.

If the arrows are widely dispersed, the estimator has high variance. If they are clustered tightly together, the estimator has low variance, even if the cluster is far from the center.

To measure total accuracy, statisticians use the mean squared error, or MSE. The MSE is the expected value of the squared errors between the estimates and the true parameter. It provides a way to see how far, on average, the estimates are from the target. A useful mathematical relationship exists here: the mean squared error is equal to the variance plus the square of the bias.

This relationship shows that a high error can be caused by either a lack of precision (high variance) or a lack of accuracy (high bias).

In many practical scenarios, scientists face a bias-variance tradeoff. This is a situation where an estimator might have a little bit of bias but a much lower mean squared error. Sometimes, allowing a small amount of bias can lead to more reliable results with fewer extreme outliers. An ideal situation is to find an unbiased estimator with very low variance. The most sought-after version of this is the minimum-variance unbiased estimator, known as the MVUE.

Researchers must decide which properties are most important based on the specific conditions of their study.

Finally, we can evaluate how an estimator behaves as more data becomes available through consistency. A consistent estimator is one where the sequence of estimates converges toward the true quantity as the sample size grows without bound. This means that as you collect more information, the probability of your estimate being correct increases.

By understanding these complex relationships between bias, variance, and error, statisticians can build better models for everything from genetics to predictive machine learning.