A computer needs memory that is fast, large and cheap, but no single technology offers all three. The memory hierarchy solves this by combining several layers. Small, very fast memory sits close to the CPU, and larger, slower and cheaper storage sits further away. The system keeps the data it uses most in the fast layers.
This guide explains each layer, how fast and large it typically is, and why the hierarchy works so well.
The layers of the memory hierarchy
| Level | Typical size | Typical access time | Volatile? |
|---|---|---|---|
| CPU registers | Hundreds of bytes to a few KB | Under 1 nanosecond | Yes |
| L1 cache | Tens of KB per core | About 1 ns | Yes |
| L2 cache | Hundreds of KB to a few MB per core | A few ns | Yes |
| L3 cache | Several MB to hundreds of MB, shared | Around 10 to 40 ns | Yes |
| Main memory (RAM) | GBs to TBs | Around 50 to 100 ns | Yes |
| SSD (NVMe) | Hundreds of GB to many TB | Tens of microseconds | No |
| Hard disk drive | Several TB | Several milliseconds | No |
| Archive (tape, cloud cold storage) | Practically unlimited | Seconds to hours | No |
These figures vary between systems, but the pattern is constant: each step down is slower, larger and cheaper per byte.
1. Registers
Registers live inside the CPU core and hold the values the processor is working on right now. There are very few of them, so compilers work hard to use them well.
2. Cache memory
Cache is fast SRAM placed on the CPU chip. It keeps copies of recently used data and instructions from RAM.
- L1 is the smallest and fastest, split into instruction and data caches.
- L2 is larger and slightly slower.
- L3 is shared by all cores and helps them exchange data.
When the CPU finds data in cache, it is a cache hit. Otherwise, it is a miss, and the CPU waits for RAM.
3. Main memory (RAM)
Main memory, also called primary memory, holds the programs and data currently in use. It is built from DRAM, which is much denser and cheaper than SRAM but slower. RAM is volatile: it loses its contents when the power goes off.
Related types of memory include ROM and its successors, such as EEPROM and flash. These keep data without power and store firmware, such as the UEFI or BIOS.
4. Secondary storage
Secondary storage keeps data permanently:
- SSDs use flash memory and have no moving parts. NVMe SSDs connect over PCIe and are very fast.
- Hard disk drives store data magnetically on spinning platters. They are slower, but cheap per terabyte.
The operating system also uses secondary storage for virtual memory, moving inactive pages out of RAM. Learn more in our guide to virtual memory.
5. Tertiary and archive storage
Tape libraries, optical media and cloud “cold” storage tiers hold backups and archives. Access is slow, but the cost per terabyte is very low, and offline copies protect against ransomware.
Why the hierarchy works: locality
Programs do not access memory randomly. They show locality of reference:
- Temporal locality: data used now is likely to be used again soon, such as a loop counter.
- Spatial locality: data near recently used data is likely to be used next, such as the next element in an array.
Because of locality, a small fast cache can serve most requests. As a result, the system runs almost as fast as if all memory were cache, at a fraction of the cost.
Advantages and limits
Advantages:
- high performance at a reasonable cost;
- large total capacity;
- permanent storage for data that must survive power loss.
Limits:
- cache misses and page faults still slow programs down;
- managing several layers adds complexity for hardware and the OS;
- writing performance-critical code often requires attention to memory layout.
Related guides
Sources
Running Linux servers in production? Our Linux server hardening & security audit checks configuration, patching, SSH and logging. You can also use our free Linux server hardening checklist, or contact us.
