Multivariate normal distribution

Sometimes we track many things at once.

You might look at height and weight together. Most things stay near a middle value. They cluster around that center.

These things can also be linked. One thing might change when the other does. This helps us see how they move. We can even see shapes like an ellipse. This shape shows how the things group together. It is a way to find patterns in many parts.

Sometimes we track many things at once.

You might look at height and weight together. We call this a multivariate normal distribution. It is a way to study many parts at the same time.

Most things stay near a middle value. These values cluster around a center point. This center is called the mean.

The parts can also be linked. One thing might change when the other does. This link is called correlation.

When we plot these points, we see shapes. In two dimensions, these shapes look like ellipses. An ellipse is a stretched circle. The shape shows how the parts group together. A tall, thin ellipse means the parts are closely linked. A round shape means they are not.

We use a special tool called a covariance matrix to describe these parts. This matrix tells us the mean and how the parts move together. If the matrix is not full rank, the shape is called degenerate. This means the points do not spread out in all directions. Scientists use these math tools to find patterns in many different sets of data.

Sometimes we want to study many different things at once. Imagine you are measuring the height and weight of a large group of people. Instead of looking at just one number, you look at how they change together. This is what we call a multivariate normal distribution. It is a way to describe a set of many variables at the same time.

These variables often cluster around a middle value called the mean. This math tool helps us understand how different pieces of information fit together in a big picture.

To make this work, we use a few special tools. One important tool is called the mean vector, which shows the center for all the parts. Another tool is the covariance matrix, which tells us how the parts move together.

If one variable goes up, the covariance matrix shows if another one usually goes up too. This link between variables is known as correlation. We can also talk about a standard normal random vector. This is a special case where every part is independent and has a mean of zero.

Mathematicians have found many ways to define this idea. One way is to say that every linear combination of the parts is also normally distributed. This means if you mix the parts together in a certain way, the result still follows a normal pattern. Another way involves using a characteristic function. You can also think about the shape the data makes. In a two-dimensional space, the points form shapes called ellipses. These ellipses are like stretched-out circles that show where the data is most likely to be found.

There are different types of these distributions depending on their shape. A distribution is called non-degenerate if its covariance matrix allows it to spread out in all directions. In this case, we can use a density function to find the probability of a point. If the matrix is not full rank, the distribution is called degenerate. This means the points do not spread out in every possible direction. This often happens in specific math tasks like ordinary least squares regression. In these cases, the shape might look flatter than a normal ellipse.

This math helps us connect many ideas we already know. For example, the Mahalanobis distance measures how far a point is from the center. It is a way to find distance that accounts for how the data is spread out. We can also use these ideas to find the probability that a sample falls inside a certain area. This is similar to how we use standard deviation to see how far a single number is from its mean. By using these tools, scientists can find patterns in very complex sets of data. They can see how many different things are working together at once.

The multivariate normal distribution is a fundamental concept in probability theory and statistics. It acts as a generalization of the one-dimensional, or univariate, normal distribution. While a univariate distribution describes a single random variable, the multivariate version describes a random vector containing several variables at once. This tool is essential because it can describe sets of correlated real-valued random variables. These variables often cluster around a specific central value known as the mean.

To understand how this distribution works, we must look at its mathematical parameters. A k-dimensional random vector is defined by two main components: a mean vector and a covariance matrix. The mean vector, denoted as $\mu$, identifies the center of the distribution in k-dimensional space. The covariance matrix, denoted as $\Sigma$, describes how the different variables change in relation to one another. This matrix must be symmetric and positive semidefinite. If we take the inverse of this covariance matrix, we get the precision matrix, denoted as $\Sigma^{-1}$.

There are several specific types of multivariate normal vectors. A standard normal random vector is a special case where every component is independent. In this version, each component has a mean of zero and a unit variance of one. A centered normal random vector is another variation. It can be created by transforming a standard normal random vector using a specific matrix. A general normal random vector is formed by taking a standard normal vector and applying both a mean vector and a covariance matrix to it. This process shows that the variables are often not independent but are instead the result of applying a matrix to independent Gaussian variables.

Mathematicians use several equivalent definitions to identify these distributions. One definition states that a vector is multivariate normal if every linear combination of its components is also normally distributed. For example, if you take a constant vector and multiply it by your random vector, the resulting single value must follow a univariate normal distribution. Another definition involves the characteristic function of the vector. Additionally, the spherical normal distribution is unique because its components remain independent in any orthogonal coordinate system.

When the covariance matrix is positive definite, the distribution is called non-degenerate. In this state, the distribution possesses a density function. The shape formed by points with equal density is known as an iso-density locus. In k-dimensional space, these loci are ellipses or their higher-dimensional generalizations. This makes the multivariate normal a special case of the broader family of elliptical distributions. To measure how far a specific point is from the mean, we use the Mahalanobis distance. This distance is different from standard Euclidean distance because it accounts for the spread and correlation of the data.

In a two-dimensional, or bivariate, case, the density function is easier to visualize. The iso-density loci appear as ellipses in the plane. The principal axes of these ellipses are defined by the eigenvectors of the covariance matrix. The lengths of the major and minor semidiameters are equal to the square roots of the ordered eigenvalues. As the correlation between the two variables increases, these ellipses become more squeezed. This squeezing follows a specific line, which represents the best linear unbiased prediction of one variable given the other.

Sometimes, the distribution becomes degenerate. This occurs when the covariance matrix is not full rank, meaning it is singular. A degenerate distribution does not have a density with respect to the usual k-dimensional Lebesgue measure. This situation often appears in statistics, such as when looking at the distribution of residuals in ordinary least squares regression. To handle these cases, mathematicians may use the disintegration theorem. This allows them to define a density by restricting attention to a specific subspace where the distribution is supported.

This mathematical framework connects to many advanced fields. For instance, the log likelihood of an observed vector is a quadratic form, which is distributed as a generalized chi-squared variable. The concept of differential entropy can also be applied, measured in units called nats. Furthermore, the Kullback–Leibler divergence allows researchers to measure the difference between two multivariate normal distributions. By using these tools, scientists can calculate mutual information and understand complex dependencies in high-dimensional data sets.