Standard deviation

Standard deviation is a fundamental statistical tool used to quantify variation. It measures how much individual values in a data set deviate from the arithmetic average, or mean. In any collection of data, values rarely sit exactly on the average. Some are higher, and some are lower. A low standard deviation indicates that most data points cluster closely around the mean. Conversely, a high standard deviation shows that the values are spread across a much wider range.

To calculate the population standard deviation, mathematicians follow a precise sequence of operations. First, they determine the mean of the entire data set. Next, they calculate the deviation for each individual value by subtracting the mean from that value. These deviations are then squared to ensure all results are positive. The average of these squared deviations is called the variance. The final step is to take the square root of the variance to arrive at the standard deviation.

One major advantage of standard deviation is its relationship to the original data. While variance is measured in squared units, the standard deviation is expressed in the same units as the data itself. This makes it much easier to interpret in real-world contexts. For example, if you are measuring height in inches, the standard deviation is also in inches. This property allows researchers to use the value to describe the actual scale of the physical measurements they are studying.

When researchers cannot measure an entire population, they must use a sample. A sample is a smaller group chosen from a larger parent population. Calculating the standard deviation for a sample requires a specific adjustment known as Bessel's correction. Instead of dividing by the total number of observations, $n$, mathematicians divide by $n - 1$. This division by a smaller number produces an unbiased estimate of the population variance. Without this correction, the sample variability would likely be underestimated because the sample mean is naturally close to the sample observations.

Standard deviation is particularly powerful when applied to a normal distribution. A normal distribution is a bell-shaped curve where data is symmetrical around the mean. In such a distribution, the standard deviation follows the 68–95–99.7 rule. This rule states that approximately 68% of observations fall within one standard deviation of the mean. About 95% fall within two standard deviations, and 99.73% fall within three.

We can see this rule in action by looking at human biology. In the United States, the average height for adult men is approximately 69 inches. The standard deviation for this height is roughly 3 inches. This means most men fall between 66 and 72 inches tall. If the standard deviation were zero, every man would be exactly 69 inches.

Standard deviation is also closely related to the concept of standard error. While standard deviation measures the spread of a population, standard error measures the spread of a sample mean. The standard error is calculated by dividing the population standard deviation by the square root of the sample size. Scientists use this to determine statistical significance. By convention, an effect is often considered significant if it is more than two standard errors away from the null expectation.

Finally, it is important to note that not all mathematical distributions have a standard deviation. Some distributions have "fat tails," meaning they extend toward infinity in a way that prevents the calculation from converging. For instance, the Cauchy distribution has neither a mean nor a standard deviation. In more complex settings, such as two-dimensional data, the standard deviation can be visualized as an ellipse.