Maximum likelihood estimation

Maximum likelihood estimation, often called MLE, is a fundamental method used in statistics. It allows researchers to estimate the unknown parameters of a probability distribution. When we observe data, we often want to know the underlying rules that created it. MLE works by choosing the parameter values that make the observed data most probable. The specific value that maximizes this probability is called the maximum likelihood estimate. This method is a dominant tool for statistical inference because it is both intuitive and flexible.

The mechanism of MLE begins with a mathematical model. We treat a set of observations as a random sample from an unknown joint probability distribution. This distribution is defined by a set of parameters, which we represent as a vector. These parameters exist within a parameter space, which is a subset of Euclidean space. To perform the estimation, we evaluate the joint density of the observed data. This evaluation creates a real-valued function known as the likelihood function. If the variables are independent, this function is the product of individual density functions. The goal is to find the point in the parameter space where this function reaches its highest value.

There are several ways to solve the math required to find this maximum. If the likelihood function is differentiable, mathematicians can use a derivative test. This involves finding the points where the derivative is zero, known as the likelihood equations. In some specific cases, such as ordinary least squares for linear regression with normal errors, these equations can be solved analytically. However, for many complex models, no closed-form solution exists. In these instances, researchers must use numerical optimization to find the estimate. To ensure a point is a true maximum rather than just a local peak, they check the Hessian matrix. This matrix of second-order derivatives must be negative semi-definite to indicate local concavity.

Sometimes, the estimation process requires additional rules called constraints. The parameter space might not be a simple open set, so we must restrict the values allowed. For example, in a multivariate normal distribution, the covariance matrix must be positive-definite. Scientists handle these restrictions using the method of substitution or the method of Lagrange. The method of Lagrange uses Lagrange multipliers to solve constrained optimization problems. This can even lead to the Lagrange multiplier test, which checks if a constraint is valid. If the constraints are not binding at the maximum, the multipliers will be zero.

MLE possesses several powerful mathematical properties that emerge as the sample size increases. One key property is consistency, which means the estimate converges in probability to the true value as more data is collected. Another is equivariance, meaning the estimator behaves predictably under certain transformations. A third property is efficiency. As the sample size tends to infinity, the MLE achieves the Cramér–Rao lower bound. This means no other consistent estimator has a lower asymptotic mean squared error. This efficiency also results in the estimator having asymptotic normality.

While MLE is highly effective, it is important to understand its limitations in small samples. In finite samples, MLE does not have optimum properties, and other estimators might be more accurate. There is also a concept called second-order bias. The MLE can have a bias of order one over the sample size. Mathematicians can create a bias-corrected maximum likelihood estimator to fix this. This corrected version is more accurate by subtracting specific terms derived from the Fisher information matrix. This process can even be continued to third-order terms to improve precision.

Finally, MLE connects deeply to other statistical philosophies. In frequentist inference, MLE is a special case of an extremum estimator. In Bayesian inference, it relates closely to maximum a posteriori (MAP) estimation. If a researcher assumes a uniform prior distribution, the Bayesian estimate is identical to the MLE. This connection shows how different mathematical perspectives can arrive at the same truth. Whether using a frequentist or Bayesian approach, the likelihood function remains a central tool for understanding data.