Likelihood function

A likelihood function is a mathematical tool used to evaluate statistical models. It measures how well a specific model explains observed data. It does this by calculating the probability of seeing that data under different possible values for the model's parameters. In statistics, a parameter is a value that defines a specific characteristic of a model. For example, a parameter might represent the fairness of a coin. By testing many different parameter values, scientists can determine which one makes the observed results most plausible.

The mechanism of the likelihood function depends on the type of data being analyzed. It is constructed from the joint probability distribution of the random variables that generated the observations. When we evaluate the function using actual data points, the data becomes fixed. The function then becomes a function solely of the model parameters. For discrete distributions, we use a probability mass function. For continuous distributions, we use a probability density function. In both cases, the likelihood function treats the observed data as a fixed reality and varies the parameters to see which fit best.

There are two main ways to treat these parameters: discrete and continuous. In a discrete distribution, the parameter might represent a specific count or a fixed outcome. The likelihood is the probability that a particular outcome is observed when the true parameter value is a certain amount. In a continuous distribution, the parameter can take any value within a range. Here, the likelihood is defined using a probability density function. While a probability density function tells us about the likelihood of random variables, the likelihood function tells us about the likelihood of the parameters themselves.

History and different mathematical schools of thought have shaped how we use this tool. In the frequentist paradigm, researchers often use the likelihood to find a single best estimate. This process is called maximum likelihood estimation. In this method, the parameter that maximizes the likelihood function serves as a point estimate. Conversely, Bayesian statistics takes a different approach. Instead of looking for one single value, Bayesian researchers look for the posterior probability. This is the probability of the parameter being true given the observed data. They calculate this using Bayes' rule, which connects the likelihood to prior knowledge.

Maximum likelihood estimation relies on specific mathematical requirements called regularity conditions. For an estimate to exist, the likelihood function usually needs to be continuous on a compact parameter space. This means the range of possible values must be closed and bounded. If the function is twice continuously differentiable, mathematicians can use the Hessian matrix to find the maximum. This matrix of second partial derivatives helps identify the peak of the function. Furthermore, the Fisher information can be approximated by this Hessian matrix. This value provides an indication of the precision of the estimate, telling researchers how certain they can be about their result.

A key concept in comparing different ideas is the likelihood ratio. This is the ratio of any two specified likelihoods. It is a central part of likelihoodist statistics, which follows the law of likelihood. This law states that the degree to which data supports one parameter value over another is measured by this ratio. In frequentist inference, this ratio forms the basis for the likelihood-ratio test. This test is considered the most powerful way to compare two simple hypotheses at a given significance level. In Bayesian inference, this ratio is known as the Bayes factor. It is used to update the odds of one theory against another.

The likelihood function connects to many advanced mathematical and scientific fields. In measure-theoretic probability theory, the density function is defined as the Radon–Nikodym derivative. This allows scientists to construct likelihood functions for many types of distributions, including mixed continuous-discrete distributions. In medicine, likelihood ratios are used in diagnostic testing to assess the value of a medical test. In pure mathematics, theorems like Wilks' theorem describe the distribution of the log-likelihood ratio. These connections show that the likelihood function is not just a simple calculation, but a fundamental pillar of modern scientific reasoning.