Cache (computing)

In computing, a cache is a hardware or software component designed for speed. It stores data so that future requests for that information can be served much faster. A cache might hold the result of an earlier computation or a copy of data stored elsewhere.

Because caches are meant to be extremely fast, they must remain relatively small. This is due to an inherent trade-off in memory design. Larger capacities require larger physical sizes. This creates greater distances for electronic signals to travel. These distances cause propagation delays that slow down the system.

The mechanism of a cache relies on a pool of entries. Each entry contains specific data and an associated tag. The tag identifies which piece of data from a slower backing store is being copied. When a client, such as a CPU or web browser, needs data, it first checks the cache. If the tag matches the requested data, a cache hit occurs. The system reads the data directly from the cache, which is very fast. If the tag does not match, a cache miss occurs. The system must then perform a more expensive access to the slower backing store.

Once retrieved, the new data is typically copied into the cache. To make room, the system uses a replacement policy to remove an existing entry. One common policy is least recently used (LRU), which removes the entry accessed longest ago.

Caches manage data using different write policies to handle updates. A write-through cache performs writes synchronously to both the cache and the backing store. In contrast, a write-back cache initially writes only to the cache. This method postpones the update to the backing store until the modified content is about to be replaced. This delayed process is known as a lazy write. Because write-back caches are more complex, they must track which locations have been modified. These modified locations are marked as "dirty" to ensure they are eventually updated.

When a write miss occurs, systems also use write-miss policies. A write-allocate policy loads the data into the cache after a miss. A no-write-allocate policy writes the data directly to the backing store without loading it into the cache.

Efficiency in caching is driven by the principle of locality of reference. This concept describes how computer applications typically access data in predictable patterns. Temporal locality occurs when data that was recently requested is likely to be requested again soon. Spatial locality occurs when data stored near recently requested data is likely to be needed next. To take advantage of this, many systems use prefetching. Prefetching involves guessing where future reads will come from and requesting that data ahead of time. If done optimally, this allows the system to bypass latency altogether.

Different types of hardware utilize specialized caching structures. Central processing units (CPUs) have used caches since the 1980s. Modern high-end processors may use as many as six different types of cache. These include the instruction cache (I-cache) and the data cache (D-cache). Another specialized cache is the translation lookaside buffer (TLB). The TLB is used by the memory management unit (MMU) to record virtual-to-physical address translations. Graphics processing units (GPUs) also use caches to handle texture data and shader instructions. Digital signal processors (DSPs) have evolved to include similar cache structures to CPUs.

Caching also plays a vital role in network architecture. In information-centric networking (ICN), the network itself acts as a loosely connected network of caches. This is different from a standard proxy server because it is a network-level solution. ICN requires very fast and lightweight eviction policies because cache states change rapidly. Some network caches use a variant called time-aware least recently used (TLRU). This algorithm uses a "time to use" (TTU) timestamp. The TTU allows administrators to regulate storage based on the usability time of the content.

Understanding caches helps explain the balance between latency and throughput. Latency is the delay before a transfer begins, while throughput is the amount of data moved. A large resource, like DRAM, can incur significant latency. For a modern 4 GHz processor, reaching DRAM can take hundreds of clock cycles. Caches mitigate this by reading large chunks of data at once. This can also increase throughput by assembling small transfers into larger, more efficient requests. This connection between small, fast memory and large, slow storage is fundamental to modern computing.