Why NUMA and Memory Locality Matter in Low-Latency Trading Systems | Trading Engineering — Deep Dive #4

Why NUMA and Memory Locality Matter in Low-Latency Trading Systems

Trading Engineering — Deep Dive #4

In low-latency trading systems, CPU performance is only part of the story.

Where the data is located can be just as important.

Modern servers may use multiple CPU sockets and NUMA (Non-Uniform Memory Access) architectures. In such systems, memory access does not always have the same cost.

Even within a single server, the latency of a memory access can depend on:

Which CPU is accessing which memory.

This makes NUMA and Memory Locality important considerations when designing high-performance trading systems.

---

cpu memory access

 

1. What Is NUMA?

NUMA stands for Non-Uniform Memory Access.

The name describes the fundamental characteristic of the architecture:

Memory access cost is not necessarily uniform.

In a simple single-CPU system, we might imagine:

CPU
 ↓
Memory

The CPU accesses the system memory through a relatively simple path.

A multi-socket server can look different:

        CPU 0                         CPU 1
          │                            │
      Local Memory                 Local Memory
          │                            │
          └────────── Interconnect ───┘

Memory physically associated with CPU 0 is typically considered local memory for CPU 0.

Memory associated with CPU 1 is local to CPU 1.

Therefore:

CPU 0 → Memory 0    Local Access

CPU 0 → Memory 1    Remote Access

The two accesses can follow different paths through the system.

---

2. Why Does Memory Locality Matter?

Low-latency systems are not only about reducing CPU instructions.

The CPU also needs to access data efficiently.

A simplified data path looks like:

CPU
 ↓
Cache
 ↓
Memory

If the required data is already available in a nearby cache, the CPU may avoid accessing main memory altogether.

When memory access is required, local memory is generally preferable to remote memory.

For example:

Local Access

CPU 0
  │
  ↓
Memory 0

versus:

Remote Access

CPU 0
  │
  ↓
Interconnect
  │
  ↓
Memory 1

Remote memory access introduces additional communication through the system interconnect.

In high-frequency or high-throughput workloads, these additional costs can become relevant.

---

3. CPU Affinity and NUMA Should Be Considered Together

In Deep Dive #3, we discussed CPU Affinity and Core Pinning.

But fixing a thread to a CPU core is only part of the problem.

Consider:

CPU Core 0
   │
   └── NUMA Node 0
          │
       Memory 0

A thread running on Core 0 may still repeatedly access memory located on another NUMA node.

Therefore, CPU placement alone does not guarantee good locality.

A low-latency system should consider:

Thread
   ↓
CPU Core
   ↓
NUMA Node
   ↓
Memory

as a connected system.

---

4. Think About the Physical Relationship Between CPU and Memory

A simplified multi-socket server might look like this:

          NUMA Node 0              NUMA Node 1

        ┌─────────────┐          ┌─────────────┐
        │   CPU 0     │          │   CPU 1     │
        │  Core 0~N   │          │  Core 0~N   │
        └──────┬──────┘          └──────┬──────┘
               │                        │
          Memory 0                  Memory 1
               │                        │
               └────── Interconnect ────┘

If an application running primarily on NUMA Node 0 accesses Memory 0, the data path remains local.

If it frequently accesses Memory 1, the request may have to cross the interconnect.

This is why CPU placement and data placement should be designed together.

---

5. What Does This Mean for a Trading System?

Consider a simplified trading architecture:

Market Data
     ↓
Order Book
     ↓
Strategy
     ↓
Risk Check
     ↓
Order Manager
     ↓
DMA / FEP

If these components are distributed across different NUMA nodes, data may repeatedly cross NUMA boundaries.

For example:

NUMA Node 0

Market Data
     ↓
Order Book
     ↓
Strategy
     ↓
Risk
     ↓
Order Manager

A different NUMA node could then be used for:

NUMA Node 1

Monitoring
Logging
Administration
Other Services

This type of separation can help keep latency-sensitive processing within a locality domain.

However, the optimal architecture depends on the actual workload and hardware topology.

---

6. The NIC Matters Too

For trading systems, looking only at CPU and memory is not enough.

The network interface card (NIC) also has a physical relationship with the NUMA topology.

For example:

             NUMA Node 0
        ┌──────────────────┐
        │ CPU              │
        │ Memory           │
        │                  │
        │ NIC              │
        └──────────────────┘

If the NIC and the processing CPU are located within the same NUMA domain, the data path can be more local.

Compare this with:

CPU 0
 │
 ↓
NUMA 0
 │
Interconnect
 │
 ↓
NUMA 1
 │
NIC

In this case, network data may have to cross the NUMA interconnect before reaching the CPU.

Therefore, for high-performance network applications, it is useful to understand the complete topology:

CPU ↔ Memory ↔ NIC

---

7. Memory Allocation Matters

In a NUMA system, simply calling malloc() does not mean that the developer has explicitly controlled where the physical memory will reside.

Actual memory placement can depend on the operating system and NUMA memory policies.

Linux provides several mechanisms for inspecting and controlling NUMA behavior.

Common tools and libraries include:

numactl
libnuma

For example:

numactl --cpunodebind=0 --membind=0 ./trading_app

Conceptually, this requests that the application run on NUMA Node 0 and allocate memory from Node 0.

In production environments, however, CPU topology, other processes, memory pressure, and system-wide resource allocation must also be considered.

---

8. First-Touch and Memory Placement

NUMA systems also make the concept of First-Touch important.

A simplified view is:

Memory Allocation
       ↓
Physical Page Placement
       ↓
Influenced by First Access

For example:

buffer = malloc(size);

/* CPU 0 */
initialize(buffer);

If the memory is subsequently accessed heavily by CPU 1, the application's actual locality may not match what the developer originally intended.

For large memory regions, understanding how pages are placed can therefore be important.

---

9. Shared Memory Is Also Affected by NUMA

Shared Memory is frequently used in financial systems for high-performance inter-process communication.

For example:

Market Data Process
        │
        ↓
   Shared Memory
        │
        ↓
Strategy Process

Shared Memory can avoid some forms of data copying, but that does not automatically make the entire data path optimal.

If multiple processes on different NUMA nodes repeatedly access and modify the same data, additional memory traffic and cache-coherency traffic can occur.

Therefore:

Shared Memory does not automatically mean maximum performance.

A more accurate principle is:

Shared Memory + Good CPU/Memory Locality

---

10. NUMA and CPU Cache Are Connected

In Deep Dive #1, we discussed CPU Cache.

NUMA should not be viewed as an independent topic.

A simplified hierarchy looks like:

Thread
  ↓
CPU Core
  ↓
L1 / L2 Cache
  ↓
L3 Cache
  ↓
Local Memory
  ↓
Remote Memory

The further data is from the CPU executing the workload, the more complex the access path can become.

Therefore, performance optimization should not simply focus on CPU frequency or instruction count.

We should consider the complete path:

Thread Placement
       ↓
CPU Core
       ↓
Cache Locality
       ↓
Memory Locality
       ↓
NUMA Locality
       ↓
Network Locality

---

11. Common NUMA Problems

1. Remote Memory Access

A CPU frequently accesses memory belonging to another NUMA node.

2. Poor CPU Placement

A thread runs on one NUMA node while the data it frequently accesses resides on another.

3. Excessive Data Sharing

Threads on different NUMA nodes frequently modify the same data.

4. Excessive Cross-NUMA Communication

Multiple processing stages repeatedly move data between NUMA nodes.

These situations can reduce the benefits of a carefully optimized low-latency architecture.

---

12. How Should NUMA Be Measured?

NUMA optimization should also begin with measurement.

First, understand the server topology.

Linux provides tools such as:

lscpu
numactl --hardware
numastat

Application-level metrics can also be useful:

• CPU utilization
• CPU migration
• Cache misses
• Memory bandwidth
• Memory latency
• NUMA hit / miss
• Remote memory access
• Context switches
• IPC
• P50 latency
• P95 latency
• P99 latency
• P99.9 latency
• Throughput

For trading systems, it is particularly important to examine tail latency rather than relying only on average latency.

---

13. Basic Principles of NUMA Optimization

Not every system requires complex NUMA-aware optimization.

However, high-performance systems can benefit from several basic principles.

Keep CPU and Data Close

Thread
  ↓
CPU
  ↓
Local Memory

Reduce Cross-NUMA Communication

Avoid unnecessary movement of data between NUMA nodes.

Define Data Ownership

Allow each thread or NUMA node to primarily manage the data it owns.

Minimize Shared Mutable Data

Avoid unnecessary sharing of frequently modified data.

Understand NIC Topology

Know which NUMA node the network interface is attached to.

Benchmark Before and After

Validate changes using real workloads rather than assumptions.

---

14. Think of CPU, Memory and NIC as One System

In low-latency trading systems, it is useful to view CPU, memory, and networking as one continuous data path.

A simplified model is:

             CPU
              │
              ↓
            Cache
              │
              ↓
           Memory
              │
              ↓
            NIC
              │
              ↓
          Network

The actual architecture is much more complex.

But the key question is simple:

Where does the data travel inside the system?

During that journey, we need to ask:

• Does the thread move to another CPU?
• Does cache locality break?
• Does the data cross a NUMA boundary?
• Does the CPU access remote memory?
• Is the NIC attached to another NUMA node?
• Is unnecessary data copying occurring?

---

15. FontesFintech Engineering Approach

Low-latency performance is rarely determined by one isolated technology.

CPU Affinity does not automatically make a system faster.

Shared Memory does not automatically make a system faster.

Similarly, NUMA awareness alone does not guarantee lower latency.

The important thing is to understand the complete data path.

Network
   ↓
NIC
   ↓
CPU Core
   ↓
Cache
   ↓
Memory
   ↓
Application

 

In a multi-socket system, NUMA topology becomes another layer of that path.

Therefore, when designing low-latency systems, we consider:

Thread Placement
→ CPU Locality
→ Cache Locality
→ Memory Locality
→ NUMA Locality
→ Network Locality

as one continuous system.

---

Conclusion

NUMA is not simply a hardware architecture concept.

In low-latency trading systems, where the CPU gets its data from can affect latency, throughput, and latency predictability.

Once CPU Affinity and Core Pinning are introduced, the next natural question is:

Where is the memory that this CPU is accessing?

That question leads directly to NUMA and Memory Locality.

Ultimately, the principle is:

Right Thread. Right Core. Right Memory.

And when networking is included:

Right Thread. Right Core. Right Memory. Right NIC.

---

Next: Trading Engineering — Deep Dive #5

epoll vs. Busy Polling in Low-Latency Trading Systems

The next article examines how Linux applications wait for network events, and why the choice between event-driven I/O and Busy Polling can matter in low-latency trading systems.