How to Measure End-to-End Latency in Trading Systems | Trading Engineering #10
How to Measure End-to-End Latency in Trading Systems
Trading Engineering Series — 10
When discussing low-latency trading systems, one of the most frequently mentioned terms is latency.
Faster CPUs, optimized networks, reduced memory copies, and minimized lock contention can all contribute to better performance.
But one important question remains:
How fast is the actual order processed from end to end?
A specific function may execute in a few microseconds, but that does not necessarily mean the entire trading system is fast.
To understand real-world performance, we need to measure End-to-End Latency.
This article explains how to define, measure, and analyze latency in a low-latency trading system.
1. Latency Is Not a Single Number
Latency in a trading system can be divided into multiple stages.
For example:
Strategy → Risk Check → Order Manager → DMA/FEP → Network → Exchange
Each stage introduces its own latency.
These may include:
- Strategy Processing Latency
- IPC Latency
- Risk Check Latency
- Order Manager Latency
- FEP Processing Latency
- Network Latency
- Exchange Response Latency
Therefore, simply saying:
“Our system has 10μs latency.”
does not tell us enough.
The important question is:
10μs measured from where to where?
2. What Is End-to-End Latency?
End-to-End Latency represents the total time required for data to travel through the defined system path.
For an algorithmic trading system, the path might look like:
Market Data Received
↓
Strategy Decision
↓
Pre-Trade Risk Check
↓
Order Generation
↓
DMA/FEP
↓
Network Transmission
↓
Exchange
If execution feedback is included, the path may continue:
Market Data → Order → Exchange → Execution Response
The exact measurement boundary should be defined according to the purpose of the system and the business requirement.
The key is to clearly define where the measurement starts and where it ends.
3. Why Is End-to-End Measurement Important?
Even if every individual component is fast, the entire system may not be fast.
For example:
- Strategy: 5μs
- Risk Check: 3μs
- Order Manager: 4μs
- FEP: 5μs
It may appear that the total latency is:
5 + 3 + 4 + 5 = 17μs
But the actual system may also include:
- IPC
- Memory Copy
- Lock Contention
- Context Switching
- Queueing
- Scheduling
- Network Processing
As a result, the real End-to-End Latency can be significantly higher.
This is why the important question is not simply:
“How fast is each function?”
but rather:
“How long does it take for the data to travel through the entire system?”
4. Identify the Critical Path
Not every operation in a trading system is equally important.
The processing path that directly affects order execution can be considered the Critical Path.
For example:
Market Data → Strategy → Risk → Order Manager → DMA/FEP → Exchange
This path directly affects trading latency.
Other operations, such as:
- Logging
- Statistics
- Monitoring
- Database Storage
may not need to be executed synchronously on the Critical Path.
Where appropriate, they can be moved to an asynchronous path:
Trading Path
→ Order Processing
Async Path
→ Logging
→ Monitoring
→ Statistics
→ Storage
Separating these workloads can help reduce unnecessary latency on the order-processing path.
5. Where Should Timestamps Be Recorded?
To measure latency, timestamps need to be recorded at meaningful points in the data path.
For example:
T1: Market Data Received
T2: Strategy Decision
T3: Risk Check Completed
T4: Order Generated
T5: FEP Processing Completed
T6: Network Send
T7: Exchange Response
We can then calculate individual stages:
Strategy Latency = T2 - T1
Risk Latency = T3 - T2
Order Processing = T4 - T3
FEP Latency = T5 - T4
Network-related Latency = T6 - T5
And the overall latency:
End-to-End Latency = T7 - T1
This allows us to identify where latency is actually being introduced.
6. Timestamping Itself Has a Cost
There is an important consideration when instrumenting a low-latency system.
The act of measuring latency can itself affect system performance.
If timestamps are recorded excessively, the measurement code may introduce additional CPU instructions and memory activity into the critical path.
This is especially important when measuring very small latency values.
Therefore:
Measurement is necessary, but measurement itself should not become a bottleneck.
Instrumentation should be carefully designed and the results should be interpreted in the context of the production environment.
7. Average Latency Is Not Enough
Looking only at average latency can be misleading.
For example:
Average: 10μs
might actually represent:
8μs
9μs
10μs
10μs
11μs
12μs
150μs
Most messages are processed quickly, but some experience a significant delay.
For latency-sensitive trading systems, these occasional delays can be important.
Therefore, latency analysis should consider:
- Average
- Minimum
- Maximum
- P50
- P90
- P95
- P99
- P99.9
In particular, Tail Latency deserves careful attention.
8. Why Tail Latency Matters
The most interesting part of a low-latency system is sometimes not its average performance, but its worst-performing fraction.
For example:
P50 = 8μs
P99 = 15μs
P99.9 = 80μs
This means most orders are processed quickly, but a small percentage experience significantly higher latency.
Possible causes include:
- Lock Contention
- Context Switching
- Cache Misses
- Memory Allocation
- CPU Scheduling
- Queueing
- Network Retransmission
- Interrupt Processing
- System Load
Therefore, improving average latency does not necessarily mean the overall system has improved.
A system with a slightly higher average but significantly better tail latency may provide more predictable behavior in production.
9. Latency Changes Under Load
A system that achieves 10μs in a lightly loaded test environment does not necessarily maintain 10μs under real market conditions.
As market data and order volume increase:
- Queues may grow
- CPU utilization may increase
- Cache misses may increase
- Lock contention may increase
- Memory bandwidth pressure may increase
Performance testing should therefore include multiple load levels.
For example:
Low Load
→ Normal market conditions
Normal Load
→ High but typical activity
Peak Load
→ Significant increase in trading volume
Stress Load
→ Near-system-limit conditions
The important question is not only:
“How fast is the system?”
but also:
“How does latency behave as system load increases?”
10. Optimization Starts with Measurement
One of the most dangerous approaches to performance optimization is optimizing without measuring.
For example, we might assume:
“memcpy() must be the bottleneck.”
or:
“The Lock must be causing the delay.”
or:
“The network must be the problem.”
These assumptions may be wrong.
The actual bottleneck could be somewhere else entirely.
A more reliable optimization cycle is:
Measure
↓
Identify
↓
Redesign
↓
Benchmark
↓
Measure Again
Performance optimization is not a one-time activity.
It is an iterative engineering process.
11. Looking Back at the Previous Articles
The topics covered throughout this Trading Engineering series are not isolated technologies.
They are connected through the same data path.
For example:
Market Data
↓
FEP
↓
IPC / Shared Memory
↓
Strategy
↓
Lock / Synchronization
↓
Risk Check
↓
Order Manager
↓
Memory Copy
↓
DMA / FEP
↓
Network
↓
Exchange
Each stage contributes to the final End-to-End Latency.
This means that building a low-latency trading system is not simply about selecting one “fast” technology.
It requires understanding the entire data path, measuring the actual bottlenecks, and optimizing the components that matter.
12. Our Approach at FontesFintech
At FontesFintech, we do not evaluate trading-system performance based on the speed of a single component.
We focus on the End-to-End Data Path.
Our engineering considerations include:
- End-to-End Latency
- Critical Path
- Memory Copy
- IPC
- Shared Memory
- Ring Buffer
- Lock Contention
- CPU Cache
- Network Processing
- Message Parsing
- Queueing
- Fault Recovery
The goal is to understand how data moves through the entire system and identify where optimization can provide meaningful results.
Ultimately, the important question is not:
“Which technology are we using?”
It is:
“How quickly and reliably can an actual order reach its destination?”
Conclusion
Latency in a low-latency trading system is not simply a single number.
We need to understand:
- Where the measurement starts
- Where it ends
- Which path is being measured
- Under what system load
- How the latency is distributed
- How the Tail Latency behaves
Most importantly, we need to take an End-to-End perspective.
Measure the Path.
Find the Bottleneck.
Optimize the Critical Path.
Throughout this first Trading Engineering series, we have explored many of the fundamental elements of low-latency trading systems:
FEP
DMA
Algorithmic Trading
Shared Memory
Lock Contention
Memory Copy
End-to-End Latency
This concludes the first part of the series.
In the next phase, we will go one level deeper — into the underlying technologies that actually determine system latency at the CPU, operating-system, and network-stack level.
Topics will include:
CPU Cache
Cache Line
CPU Affinity
NUMA
epoll & Busy Polling
NIC Offload
Kernel Bypass
TCP / UDP
This is where we move from understanding the architecture of low-latency trading systems to understanding how the underlying system actually executes them.
Measure First. Optimize with Evidence.