Why CPU Cache Matters in Low-Latency Trading Systems | Trading Engineering — Deep Dive #1

Why CPU Cache Matters in Low-Latency Trading Systems

Trading Engineering — Deep Dive #1

In previous articles, we discussed Shared Memory, Lock Contention, Memory Copy, and End-to-End Latency.

These topics all lead to a deeper question:

How does the CPU actually access data?

In modern low-latency systems, performance is often limited not by arithmetic operations, but by how efficiently data moves through the CPU memory hierarchy.

This is where CPU Cache becomes important.

For low-latency trading systems, understanding CPU Cache is not an advanced optimization technique.

It is part of the fundamental system architecture.

---

1. Why Main Memory Is Not Enough

Modern CPUs can execute instructions extremely quickly.

However, accessing DRAM is relatively slow.

A simplified view:

StorageApproximate Access Time

CPU Register	~1 cycle
L1 Cache	Few cycles
L2 Cache	Tens of cycles
L3 Cache	Tens to hundreds of cycles
DRAM	Hundreds of cycles

The CPU can execute billions of instructions per second.

If the CPU waits for DRAM access repeatedly, processing stalls occur.

For latency-sensitive trading systems, these delays accumulate.

This is why modern CPUs use multiple cache levels.

---

2. Cache Hierarchy

A simplified CPU structure:

CPU Core
   ↓
L1 Cache
   ↓
L2 Cache
   ↓
L3 Cache
   ↓
Main Memory (DRAM)

L1 Cache

• Smallest
• Fastest
• Usually private to each core

L2 Cache

• Larger
• Slightly slower
• Usually private per core

L3 Cache

• Shared across CPU cores
• Larger capacity
• Higher latency

The closer the data is to the CPU core, the faster it can be processed.

---

3. Cache Line: The Real Unit of Data Movement

One common misunderstanding is:

"The CPU reads only the variable it needs."

This is not exactly true.

CPUs usually move data in units called:

Cache Lines

A typical cache line size is:

64 bytes

For example:

int price;

Even if only 4 bytes are required, the CPU may load an entire 64-byte cache line.

This behavior is important because neighboring data can influence performance.

Data layout matters.

---

4. Why Cache Misses Are Expensive

Suppose the CPU needs data that is not in cache.

The CPU may need to:

L1 → L2 → L3 → DRAM

This process is called:

Cache Miss

Frequent cache misses increase latency.

Typical causes include:

• Large data structures
• Random memory access
• Poor data locality
• Frequent memory allocation
• Excessive data sharing

For trading systems processing millions of messages, cache misses can become significant.

---

5. Data Locality

Data locality means:

Keep frequently accessed data close together.

Example:

struct Order {
    int qty;
    int price;
    int side;
};

This structure is cache-friendly because related fields are close together.

Poor layout:

qty -> memory A
price -> memory B
side -> memory C

may require multiple cache accesses.

Good locality reduces cache misses.

---

6. Cache Coherency

Modern trading systems use multiple CPU cores.

Suppose:

Core 1 updates:

position += 1;

Core 2 reads:

position

The CPU must ensure all cores see consistent data.

This mechanism is called:

Cache Coherency

Maintaining coherency requires communication between cores.

This communication has a cost.

When many cores repeatedly update shared data, latency can increase.

---

7. False Sharing

False Sharing is a subtle performance problem.

Example:

struct SharedData {
    int counterA;
    int counterB;
};

Thread A updates:

counterA

Thread B updates:

counterB

Logically independent.

But if both variables reside in the same 64-byte cache line:

Core A ↔ Core B

Cache ownership repeatedly moves.

This creates unnecessary cache traffic.

The result:

• Increased latency
• Reduced scalability
• Higher CPU overhead

False Sharing is often overlooked in multithreaded systems.

---

8. Cache and Ring Buffers

Ring Buffers are commonly used in trading systems.

Example:

Producer
↓
Ring Buffer
↓
Consumer

If producer and consumer access nearby memory efficiently:

• Better cache utilization
• Fewer cache misses
• Less synchronization overhead

This is one reason Ring Buffers are popular in high-throughput systems.

---

9. Structure Alignment

C structures may contain padding:

struct Example {
    char type;
    int qty;
};

Compiler-generated alignment improves CPU access efficiency.

Poor alignment can increase memory access cost.

Understanding:

• Alignment
• Padding
• Cache Lines

can improve data structures.

---

10. CPU Cache and Trading Systems

A simplified trading path:

Market Data
↓
FEP
↓
Shared Memory
↓
Strategy
↓
Risk Check
↓
Order Manager
↓
DMA/FEP

At every stage:

• Which data is frequently accessed?
• Which data is shared?
• Which cache lines move between cores?
• Where do cache misses occur?

These questions are often more important than:

"Which CPU is faster?"

---

11. Practical Engineering Principles

For low-latency systems:

Prefer:

• Small structures
• Cache-friendly layout
• Sequential memory access
• Clear data ownership
• Single Writer design
• Ring Buffers

Avoid:

• Excessive sharing
• Random memory access
• Large objects
• Unnecessary allocations
• False Sharing

The objective is simple:

Keep important data close to the CPU that needs it.

---

12. Our Approach at FontesFintech

At FontesFintech, we do not view memory as an abstract concept.

We view it as:

Data movement inside a CPU system.

Our design considerations include:

• Cache-aware structures
• Shared Memory
• Ring Buffers
• Data locality
• Clear ownership
• Synchronization minimization
• Reduced cache traffic
• End-to-End latency analysis

Low-latency performance is not only about algorithms.

It is also about:

How efficiently the CPU can reach the required data.

---

Conclusion

CPU Cache is one of the foundations of low-latency system design.

The CPU does not simply execute instructions.

It continuously moves data through:

L1 → L2 → L3 → DRAM

The faster the required data can be reached:

• The lower the latency
• The higher the throughput
• The more predictable the system

Understanding CPU Cache helps us understand:

• Memory Copy
• Shared Memory
• Lock Contention
• False Sharing
• Ring Buffers
• Multithreaded Performance

And this leads to our next topic:

False Sharing in Multithreaded Trading Systems

Because in low-latency engineering:

Data placement is just as important as code execution.