Data analysis

Data analysis is the systematic process of inspecting, cleansing, transforming, and modeling data. The primary goal is to discover useful information, inform conclusions, and support decision-making. In the modern business world, this process helps organizations operate more effectively. It moves decision-making away from mere guesswork and toward a more scientific approach. This field encompasses many diverse techniques used across business, science, and social science domains.

The process begins with establishing specific data requirements. Analysts must determine what information is necessary to answer a specific question. The entity being studied is called an experimental unit, such as a person or a population. Data can be numerical or categorical, which means it uses text labels for numbers. Analysts collect this data from many sources. These include sensors like traffic cameras and satellites, or through interviews and online downloads.

Once gathered, the data must undergo integration and cleaning. Data integration involves organizing raw data into a structured format, like rows and columns in a spreadsheet. However, raw data is often incomplete or contains errors. Data cleaning is the process of preventing and correcting these mistakes. Analysts perform tasks like record matching and deduplication to remove repeated entries. They may also use outlier detection to remove numbers that were likely entered incorrectly. For text data, they might use spell checkers to fix typos.

After cleaning, analysts perform exploratory data analysis, also known as EDA. This stage focuses on discovering new features and patterns within the datasets. Analysts often use descriptive statistics to characterize the data. These include the mean, which is the average, the median, and the standard deviation. Data visualization is also used here to examine the information in a graphical format. This helps the analyst gain deeper insights before moving to more complex mathematical modeling.

Mathematical modeling is the next stage of the process. Analysts apply formulas and algorithms to identify relationships between different variables. They look for correlation, which shows if variables move together, and causality, which shows if one thing causes another. A common tool is regression analysis. This is used to see if a change in an independent variable, like advertising, explains a change in a dependent variable, like sales. The goal is to create a model where the mathematical error is as small as possible.

In 1961, the statistician John Tukey provided a formal definition of data analysis. He described it as a set of procedures for analyzing data and techniques for interpreting results. He also included the ways to plan data gathering to make the analysis more precise or accurate. This definition highlighted the importance of the "machinery" of mathematical statistics. Today, this work has branched into specialized areas. Data mining uses statistical modeling for predictive purposes, while business intelligence focuses on aggregating business information.

To communicate findings, analysts use data visualization to create clear messages. Stephen Few described eight types of quantitative messages that can be communicated. These include time-series, which tracks a variable over time, and ranking, which orders categories. Part-to-whole messages use pie charts to show ratios. Deviation compares actual results against a reference, like a budget. Frequency distributions use histograms to show how often values occur. Correlation uses scatter plots, while geographic messages use maps.

Finally, analysts often use structured principles to ensure their work is thorough. McKinsey and Company identified the MECE principle for breaking down complex problems. MECE stands for "Mutually Exclusive and Collectively Exhaustive." This means each part of the analysis must be separate from the others, but all parts together must cover the entire problem. For example, total profit can be broken down into revenue and cost. This ensures no information is overlapping and no important piece is missing.