Deep learning

Deep learning is a specialized branch of machine learning.

It uses multilayered neural networks to perform complex tasks. These tasks include classification, regression, and representation learning. The field draws inspiration from biological neuroscience. It involves stacking artificial neurons into layers to process data. The adjective "deep" describes the number of layers in the network. These layers can range from three to several thousand. Deep learning is essential for modern technology. It powers fields like computer vision and speech recognition. It also assists in drug design and climate science.

The mechanism of deep learning relies on a hierarchy of layers. This hierarchy transforms input data into abstract representations. For example, consider an image recognition model. The raw input is often a tensor of pixels. The first representational layer identifies basic shapes like lines or circles. The second layer encodes arrangements of these edges. The third layer might recognize features like a nose or eyes. The fourth layer identifies the entire face. Unlike older machine learning, features are not hand-crafted by humans. The model discovers these useful features automatically from the data. This process allows the system to learn which features to place at which level.

Researchers define "depth" through the credit assignment path, or CAP. The CAP is the chain of transformations from input to output. In a feedforward neural network, the CAP depth equals the number of hidden layers plus one. For recurrent neural networks, the CAP depth can be unlimited. This is because signals may propagate through a layer more than once. While there is no universal threshold for "deep," most researchers agree that a CAP depth higher than two qualifies. A depth of two can act as a universal approximator. This means it can emulate any function. However, adding more layers beyond two helps the model extract better features effectively.

There are many different types of deep learning architectures. Convolutional neural networks, or CNNs, are used for many tasks. Transformers are another common architecture used in modern systems. Recurrent neural networks, or RNNs, have cycles in their connectivity. This differs from feedforward neural networks, which do not have cycles. Other types include deep belief networks and generative adversarial networks. Deep belief networks can be trained in an unsupervised manner. This is a major benefit because unlabeled data is more abundant than labeled data. Neural radiance fields are also a notable architecture in the field.

The history of these ideas spans many decades. The term "deep learning" was introduced by Rina Dechter in 1986. Igor Aizenberg and colleagues used the term in 2000 regarding Boolean threshold neurons. Early neural network ideas were inspired by the human brain. However, current networks are not intended to be high-quality models of biological brain function. In 1965, Alexey Ivakhnenko and Lapa published the Group method of data handling. This was the first working deep learning algorithm. It could train arbitrarily deep networks using regression analysis. In 1979, Kunihiko Fukushima introduced the Neocognitron, an early version of a CNN.

Mathematical proofs have helped define the power of these networks. The universal approximation theorem is a key concept. It concerns the capacity of networks to approximate continuous functions. In 1989, George Cybenko published the first proof for sigmoid activation functions. Kurt Hornik generalized this to multi-layer architectures in 1991. Recent work shows this also holds for non-bounded functions like the rectified linear unit, or ReLU. Kunihiko Fukushima introduced the ReLU activation function in 1969. Today, ReLU is the most popular activation function in deep learning. These mathematical foundations allow networks to handle incredibly complex data.

Deep learning connects to many broader scientific systems. It is used in bioinformatics to understand biological data. In material inspection, it helps ensure quality in manufacturing. It even powers board game programs that can challenge humans. The field also uses backpropagation to train networks efficiently. Backpropagation is an application of the chain rule. While the term was used by Rosenblatt in 1962, the modern form appeared in Seppo Linnainmaa's 1970 thesis. Today, end-to-end stochastic gradient descent is the dominant training technique. This technology continues to push the boundaries of what computers can achieve.