Prior probability

A prior probability distribution represents an assumed set of probabilities for an uncertain quantity before any new evidence is considered. In Bayesian statistics, this serves as a starting point for reasoning about the world. The uncertain quantity might be a parameter within a mathematical model or a latent variable that cannot be observed directly. For example, a prior might represent the expected proportions of voters for a specific politician before election results are tallied. Once new data is collected, mathematicians use Bayes' rule to update this prior. This process produces a posterior probability distribution, which is the updated conditional distribution of the quantity given the new evidence.

There are several ways to construct a prior distribution depending on the information available. An informative prior expresses specific, definite information about a variable. For instance, if you are predicting the temperature at noon tomorrow, you might use a normal distribution centered on today's noon temperature. A strong prior is a specific type of informative prior where the initial assumption is so dominant that new data barely changes it. Conversely, a weakly informative prior provides only partial information. It steers an analysis toward reasonable solutions without being too restrictive. This is often used for regularization, which keeps statistical inferences within a sensible range.

When no information is available, researchers may use an uninformative prior. These are also called objective or diffuse priors. One classic method for creating these is the principle of indifference, which assigns equal probabilities to all possible outcomes. In many parameter estimation problems, using an uninformative prior leads to results similar to conventional statistical analysis. This happens because the likelihood function from the data often provides more information than the vague prior. Some mathematicians, known as objective Bayesians, believe that certain priors are logically required by the nature of uncertainty. They argue that these priors can be found through mathematical principles rather than personal opinion.

Edwin T. Jaynes was a major figure in the objective Bayesian school. He used the concept of symmetry to justify certain priors. For example, if a ball is hidden under one of three cups labeled A, B, and C, the most logical prior is a uniform distribution where each cup has a 1/3 probability. Jaynes argued that if you swap the labels, your prediction should not change. This invariance principle suggests that the uniform prior is the only choice that preserves the symmetry of the problem. He also championed the principle of maximum entropy, or MAXENT. This principle suggests choosing the distribution that contains the least amount of information consistent with known constraints.

Other mathematicians have developed specialized priors for specific mathematical needs. In 1932, J.B.S. Haldane proposed the Haldane prior. This is an improper prior distribution that places extreme weight on the possibility that an event will either always happen or never happen. If a researcher observes a chemical dissolving in one experiment but not in another, applying Bayes' theorem to the Haldane prior results in a uniform distribution. Harold Jeffreys also contributed by creating a systematic way to design uninformative priors, such as the Jeffreys prior for Bernoulli random variables. These mathematical tools allow for more precise handling of different types of data and uncertainty.

Advanced modeling often involves hierarchical priors. In these systems, the prior distributions of model parameters depend on their own parameters, known as hyperparameters. For example, if you use a beta distribution to model a parameter $p$ of a Bernoulli distribution, the $\alpha$ and $\beta$ values of the beta distribution are the hyperparameters. This allows for many conditional levels of distributions within a single model. Historically, choosing priors was difficult because mathematicians had to use conjugate families to ensure the posterior remained easy to calculate. However, the widespread availability of Markov chain Monte Carlo methods has made this constraint much less of a concern in modern science.

Modern applications also use priors for mechanical purposes like feature selection. Researchers like José-Miguel Bernardo introduced reference priors to maximize the information gained from data. This involves maximizing the expected Kullback–Leibler divergence between the prior and the posterior. By doing this, the researcher ensures the prior is as "least informative" as possible about the parameter. Whether through subjective judgment or objective mathematical principles, the prior remains a fundamental component of how we bridge the gap between what we assume and what we observe.