Big data

Computers hold many facts.

This pile of facts is called big data. It is very large. It is also very complex.

It grows very fast every day. This happens because of many tools. Cameras and phones make new facts. These facts can help us. They can stop crime. They can even help doctors. We use many tools to study them. It is a big job to keep them all.

Computers hold huge amounts of facts. We call this big data.

It is too big for most regular software to handle. To understand it, we look at four main parts.

First is volume. This is the amount of data. It can be many terabytes or even zettabytes. Second is variety. This means the data comes in many forms. It can be text, images, or even video. Third is velocity. This is the speed at which data is made. It often happens in real time.

Fourth is veracity. This means how much we can trust the data. If data is not reliable, it can cause risks.

Many things make big data. Phones, cameras, and sensors all collect facts. Every day, 2.5 exabytes of data are made.

This data helps us in many ways. It can help doctors find diseases. It can also help stop crime. Experts use many servers to study it. This helps them find new trends in the world.

Computers today deal with massive amounts of information. We call this collection of information big data.

It is not just about having a lot of facts. Big data is too large or complex for regular software to handle. Most normal tools cannot capture or manage it in a reasonable amount of time. Instead, scientists and businesses use special tools to find hidden patterns. This helps them understand the world in new ways.

To understand big data, we look at several important traits. The first is volume, which is the huge amount of data stored. The second is variety, meaning the data comes in many forms like text, video, or audio. Third is velocity, which is the high speed at which data is created. It often happens in real time.

A fourth trait is veracity, which means how reliable or truthful the data is. Without high veracity, the data might lead to mistakes. Finally, there is value, which is the worth of the information found.

People have been using the term big data since the 1990s. Some people say John Mashey helped make the name popular. In the past, companies built special systems for their own needs. For example, Teradata Corporation made a system in 1984 that could process data in parallel. They were the first to store and analyze 1 terabyte of data in 1992. Later, in 2007, they installed a system that could handle a petabyte of data. In 2000, a company called Seisint developed a platform to handle different types of data across many servers.

Data is growing at a very fast rate every single day. Every day, about 2.5 exabytes of data are generated.

This amount is incredibly large. Experts at IDC predicted global data would grow from 4.4 zettabytes to 44 zettabytes between 2013 and 2020. They also predict there will be 163 zettabytes by 2025. Using this data can save a lot of money. For example, US healthcare could create $300 billion in value every year by using it well. In Europe, governments could save over €100 billion through better efficiency.

Where does all this data come from? It comes from many devices we use every day. Mobile phones and cameras collect a lot of information. Many small sensors on the Internet of Things also send data. We also get data from software logs, microphones, and even aerial equipment. This data helps many different people in their jobs. Doctors use it to help prevent diseases. Governments use it to combat crime. Scientists use it to study the weather, biology, and even the stars.

Big data refers to datasets that are too large or too complex for traditional software to manage. Standard data-processing tools often struggle to capture, organize, or analyze this information within a reasonable timeframe. Because of this, big data requires specialized technologies and new forms of integration. These advanced methods help reveal insights from datasets that are massive in scale and highly diverse.

While the exact size of big data is a moving target, it often ranges from dozens of terabytes to many zettabytes. Understanding big data is essential because it allows us to find new correlations in the world. These connections can help businesses spot trends, prevent diseases, or even combat crime.

To understand how big data works, we look at its core characteristics. The first is volume, which is the sheer quantity of data generated and stored. The second is variety, referring to the different types of data, such as text, images, audio, and video. Third is velocity, which describes the high speed at which data is created and processed. This often happens in real time.

A fourth characteristic is veracity, which measures the reliability or truthfulness of the data. High veracity is necessary to ensure that analysis is accurate. Finally, there is value, which represents the worth or profitability of the information retrieved from the analysis.

Big data is often categorized by the structure of its information. It can include structured data, which is highly organized, as well as semi-structured and unstructured data. Unstructured data, such as social media posts or sensor logs, is a major focus of big data study. Analysts often use data fusion to complete missing pieces between these different types. Some researchers also discuss variability, which refers to how the formats or sources of data change. To handle these complexities, scientists use parallel computing. This involves using massively parallel software that runs across tens, hundreds, or even thousands of servers simultaneously.

There is a technical distinction between big data and business intelligence. Business intelligence typically uses applied mathematics and descriptive statistics to measure things and detect trends. It works with data that has high information density. In contrast, big data uses mathematical analysis, optimization, and inductive statistics to infer laws. This includes finding regressions, nonlinear relationships, and causal effects. Big data often works with datasets that have low information density. By using these advanced methods, analysts can perform predictions about future outcomes and behaviors.

The history of big data shows how quickly technology has evolved. The term has been used since the 1990s, and some credit John Mashey with making it popular. In 1984, Teradata Corporation marketed a parallel processing system. By 1992, they were the first to store and analyze 1 terabyte of data. In 2000, Seisint Inc. developed the HPCC Systems platform. This platform used a language called ECL to distribute data across multiple servers. In 2007, Teradata installed its first petabyte-class relational database management system. These milestones show how our ability to manage data has expanded alongside our hardware.

The scale of data growth is truly massive. Every day, approximately 2.5 exabytes of data are generated. The world's technological capacity to store information has roughly doubled every 40 months since the 1980s. According to IDC, global data volume was predicted to grow from 4.4 zettabytes to 44 zettabytes between 2013 and 2020. By 2025, this number is predicted to reach 163 zettabytes. This growth is driven by many devices, including mobile phones, cameras, and wireless sensor networks. Even aerial remote sensing equipment and Internet of Things devices contribute to this constant stream of information.

Using big data effectively can create enormous economic value. In 2011, McKinsey & Company reported that US healthcare could create over $300 billion in value annually through big data. In Europe, government administrators could save more than €100 billion in operational efficiency. Furthermore, users of personal-location data could capture $600 billion in consumer surplus. The market for these solutions is also growing rapidly. Global spending on big data and business analytics solutions was estimated to reach $215.7 billion in 2021. This massive investment helps scientists in fields like genomics, meteorology, and complex physics simulations.