Why False Sharing Matters in Multithreaded Low-Latency Systems | Trading Engineering — Deep Dive #2

Why False Sharing Matters in Multithreaded Low-Latency Systems

Trading Engineering — Deep Dive #2

Understanding CPU Cache is not enough to build a low-latency trading system.

Even when data is already in the CPU cache, performance can be significantly affected by how that data is laid out and accessed by multiple threads.

One of the most important examples is False Sharing.

False Sharing occurs when multiple threads access different variables that happen to reside on the same Cache Line, causing unnecessary cache-line movement and coherency traffic.

It can become particularly relevant in:

• Multithreaded Trading Systems
• Market Data Processing
• Order Processing
• Ring Buffers
• Shared Memory
• Producer / Consumer architectures
• High-frequency counters and statistics
• Low-Lock and Lock-Free designs

In this article, we will look at why False Sharing occurs and how it can appear in real C code.

---

1. What Is False Sharing?

Consider a simple structure:

struct SharedData {
    int producer_count;
    int consumer_count;
};

Suppose two threads update different variables:

Thread A
    producer_count++

Thread B
    consumer_count++

Logically, the threads are working with completely independent data.

However, CPUs generally transfer data between cache levels in units of Cache Lines, rather than individual variables.

A Cache Line is commonly 64 bytes on modern CPUs, although the actual size depends on the CPU architecture.

If both variables happen to reside on the same Cache Line, a problem can occur.

Cache Line
┌──────────────────────────────────────────────┐
│ producer_count │ consumer_count │ ...        │
└──────────────────────────────────────────────┘
        ↑                     ↑
     Thread A              Thread B

The two threads are modifying different variables.

But from the CPU's perspective, they are modifying the same Cache Line.

This is False Sharing.

---

2. Why Does It Hurt Performance?

Modern multicore CPUs have cache structures associated with individual cores.

Consider:

Core 0                         Core 1

Cache                          Cache
  │                              │
  └────── Same Cache Line ───────┘

When Thread A modifies producer_count, the corresponding Cache Line may need to be updated or invalidated in other cores' caches according to the CPU's cache-coherency protocol.

Then Thread B modifies consumer_count.

The same Cache Line may need to move or change state again.

The important point is:

The data is independent, but the Cache Line is not.

This can create unnecessary cache-coherency traffic and reduce scalability as the number of cores or update frequency increases.

---

3. False Sharing Can Happen Without Locks

This is especially important in low-latency programming.

False Sharing does not require a Mutex or Spinlock.

Even simple operations such as:

counter_a++;
counter_b++;

can be affected if the two counters are frequently modified by different threads and occupy the same Cache Line.

Therefore:

No Lock
    ↓
No Waiting
    ↓
Fast

is not always true.

A system can instead behave like:

No Lock
    ↓
Shared Cache Line
    ↓
Cache Line Invalidation
    ↓
Coherency Traffic
    ↓
Performance Degradation

---

4. A Simple C Example

Consider:

struct Counters {
    volatile long producer;
    volatile long consumer;
};

Thread A:

void *producer_thread(void *arg)
{
    struct Counters *c = arg;

    for (long i = 0; i < 100000000; ++i)
        c->producer++;

    return NULL;
}

Thread B:

void *consumer_thread(void *arg)
{
    struct Counters *c = arg;

    for (long i = 0; i < 100000000; ++i)
        c->consumer++;

    return NULL;
}

Each thread only modifies its own counter.

Yet if the two counters occupy the same Cache Line, False Sharing can occur.

It is also important to understand that volatile does not solve False Sharing.

volatile affects compiler optimization and memory-access semantics, but it is not a mechanism for thread synchronization or cache-coherency management.

---

5. Separating Data with Padding

A common approach is Padding.

The idea is to place frequently modified variables on separate Cache Lines.

Conceptually:

Cache Line A
┌──────────────────────────────────────────────┐
│ producer counter                            │
│ padding                                     │
└──────────────────────────────────────────────┘

Cache Line B
┌──────────────────────────────────────────────┐
│ consumer counter                            │
│ padding                                     │
└──────────────────────────────────────────────┘

With C11, alignment can be expressed using _Alignas.

For example:

#include <stdalign.h>

struct alignas(64) Counter {
    long value;
};

The exact implementation should still consider the target CPU architecture, ABI, compiler behavior, and actual Cache Line size.

The commonly used 64-byte value should not be treated as a universal architectural constant.

---

6. Data Ownership Is More Fundamental

Padding is useful, but there is a more fundamental design principle:

Clear Data Ownership.

Instead of having multiple threads continuously modify shared state, try to give each piece of frequently modified data a clear owner.

For example:

Market Data Thread
        │
        ▼
   Owns Market Data

Strategy Thread
        │
        ▼
   Owns Strategy State

Order Thread
        │
        ▼
   Owns Order State

Reducing shared writable data can help reduce not only False Sharing, but also:

• Lock Contention
• Cache Coherency Traffic
• Synchronization overhead

In low-latency systems, minimizing shared mutable state can therefore be more valuable than simply adding padding.

---

7. False Sharing in Ring Buffers

Ring Buffers are widely used in low-latency systems.

Producer
    │
    ▼
┌────┬────┬────┬────┬────┬────┐
│    │    │    │    │    │    │
└────┴────┴────┴────┴────┴────┘
                         │
                         ▼
                    Consumer

However, the metadata itself can become a source of False Sharing.

For example:

struct RingBuffer {
    uint64_t head;
    uint64_t tail;

    char buffer[1024 * 1024];
};

If the Producer frequently updates head while the Consumer frequently updates tail, placing both variables on the same Cache Line can create unnecessary coherency traffic.

A design may instead separate them:

Cache Line A
┌──────────────────────────────────────────────┐
│ head                                         │
└──────────────────────────────────────────────┘

Cache Line B
┌──────────────────────────────────────────────┐
│ tail                                         │
└──────────────────────────────────────────────┘

Cache Line C
┌──────────────────────────────────────────────┐
│ buffer                                       │
└──────────────────────────────────────────────┘

The goal is to allow the Producer and Consumer to modify their own metadata with less unnecessary Cache Line interaction.

---

8. Atomic Does Not Automatically Solve the Problem

A common assumption is:

"Then we should just use Atomic instead of Mutex."

Atomic operations are extremely useful in concurrent systems.

But Atomic does not eliminate cache-coherency costs.

For example:

atomic_fetch_add(&counter, 1);

The operation may provide atomicity, but if multiple cores repeatedly modify data located on the same Cache Line, cache-coherency traffic can still occur.

In other words:

Atomic
≠
No Cache Contention

When evaluating Atomic operations, consider:

• Cache Line ownership
• Cache Coherency
• Memory Ordering
• Core-to-Core communication
• Access patterns
• Frequency of updates

---

9. False Sharing vs. Lock Contention

These two problems can both hurt performance, but their causes are different.

Lock Contention

Thread A
   │
   ├── Lock acquired
   │
   └── Critical Section

Thread B
   │
   └── Waiting for Lock

The primary issue is waiting for a synchronization primitive.

False Sharing

Thread A                 Thread B
   │                        │
producer_count++       consumer_count++
   │                        │
   └──── Same Cache Line ───┘
                 │
          Cache Coherency

There is no need for a Lock.

The problem comes from the physical placement of independently modified data.

Understanding this distinction is important when diagnosing performance problems.

---

10. Where Can False Sharing Appear in Trading Systems?

False Sharing can appear in many parts of a trading system.

Market Data Processing

Thread 1 → received_count
Thread 2 → processed_count

Order Processing

Thread 1 → order_count
Thread 2 → cancel_count

Ring Buffer

Producer → head
Consumer → tail

Statistics

Thread 1 → buy_count
Thread 2 → sell_count
Thread 3 → reject_count

Each thread may appear to have independent data.

But the actual memory layout needs to be examined.

---

11. How Do We Detect False Sharing?

There is an important engineering principle here:

Do not add Padding simply because you suspect False Sharing.

Measure first.

A useful workflow is:

Measure
   ↓
Identify
   ↓
Verify
   ↓
Redesign
   ↓
Benchmark
   ↓
Measure Again

Useful measurements may include:

• Thread execution time
• Throughput
• CPU utilization
• Cache Misses
• Cache References
• Cache-to-Cache Transfers
• Hardware Performance Counters
• Context Switches
• Tail Latency

On Linux, CPU performance counters can be used to investigate cache-related behavior.

The important distinction is between suspecting False Sharing from source code and demonstrating its actual performance impact through measurement.

---

12. Padding Is Not Always Better

Padding should not be added indiscriminately.

For example:

struct HugeData {
    long counter;
    char padding[4096];
};

Artificially increasing the size of every object can increase memory consumption and the overall working set.

That can negatively affect cache utilization.

The goal is not:

"Separate everything as much as possible."

The goal is:

"Separate data that is actually causing unnecessary Cache Line interaction."

---

13. Practical Principles for Low-Latency Systems

The following principles are useful when designing multithreaded low-latency systems.

1. Define Data Ownership

Whenever practical, let one thread primarily own and modify a piece of data.

2. Minimize Shared Writable Data

Avoid having many threads repeatedly modify the same state.

3. Examine Hot Data Layout

Understand where frequently accessed and frequently modified variables are located in memory.

4. Consider Cache Line Boundaries

Use alignment and padding when measurement shows they are beneficial.

5. Separate Ring Buffer Metadata

Consider separating frequently modified Producer and Consumer metadata.

6. Use Atomic Operations Carefully

Atomic operations are powerful, but they do not eliminate cache-coherency costs.

7. Benchmark the Real System

Measure before and after applying an optimization.

---

14. False Sharing Is a Memory Layout Problem

When discussing system performance, we often focus on:

CPU
Network
Algorithm
Compiler

But performance can also depend heavily on how data is physically arranged in memory.

These two logical structures may look almost identical:

A B

But from the CPU's perspective:

Cache Line 1
┌──────────┬──────────┐
│    A     │    B     │
└──────────┴──────────┘

and:

Cache Line 1
┌──────────┐
│    A     │
└──────────┘

Cache Line 2
┌──────────┐
│    B     │
└──────────┘

can have very different performance characteristics under concurrent access.

This is why low-latency system design sometimes needs to consider not only the logical structure of a data structure, but also its physical memory layout.

---

15. FontesFintech's Approach

Solving False Sharing in a low-latency Trading System is not simply about adding Padding.

FontesFintech considers:

• Data Ownership
• Minimizing Shared Data
• Cache-friendly Data Layout
• Cache Line Alignment
• Minimizing False Sharing
• Ring Buffer Design
• Appropriate Use of Atomic Operations
• Minimizing Lock Contention
• Minimizing Memory Copy
• Simplifying the Critical Path
• Measuring End-to-End Latency

The key is not one particular optimization technique.

It is understanding:

Where is the data?
Who modifies it?
How frequently is it accessed?
How often does it interact with data used by another Core?

---

Conclusion

False Sharing is a performance problem that can be difficult to identify from source code alone.

Two threads may work with completely independent variables, yet still affect each other because those variables occupy the same Cache Line.

The relationship can be summarized as:

Data Layout
     ↓
Cache Line
     ↓
Cache Coherency
     ↓
Core-to-Core Traffic
     ↓
Latency

Improving low-latency performance therefore requires more than a faster CPU or a faster algorithm.

The way data is organized in memory can be just as important as the way code is executed.

In low-latency systems, where data is placed matters as much as how code is executed.

Trading Engineering — Deep Dive

Data Ownership.
Cache Awareness.
Measured Performance.

Next: CPU Affinity and Core Pinning in Low-Latency Trading Systems