Interrupts, Polling and Network Latency in Low-Latency Trading SystemsTrading | Engineering — Deep Dive #6

Interrupts, Polling and Network Latency in Low-Latency Trading Systems

Trading Engineering — Deep Dive #6

In a low-latency trading system, receiving a packet is only the beginning.

A market data packet or an order response must travel through multiple layers before the application can actually process it.

A simplified path looks like this:

NIC
 ↓
Interrupt / Polling
 ↓
Kernel
 ↓
Network Stack
 ↓
Socket
 ↓
Application

 

Each layer can introduce latency.

More importantly, the way the system detects incoming packets can significantly affect both latency and CPU utilization.

This is why understanding Interrupts, Polling and NAPI is important when designing low-latency trading systems.

 

---

1. What Happens When a Network Packet Arrives?

Let's start with a simplified example.

A market data packet arrives at the server:

Exchange
   ↓
Network
   ↓
NIC
   ↓
Linux Kernel
   ↓
Socket Buffer
   ↓
Trading Application

 

The application does not simply "receive the packet" immediately.

There are several stages between the physical arrival of the packet and the application's recv() or read() call.

Understanding these stages helps explain why different networking architectures have different latency characteristics.

---

2. Traditional Interrupt-Driven Networking

One traditional approach is interrupt-driven networking.

Conceptually:

Packet Arrives
      ↓
NIC
      ↓
Interrupt
      ↓
CPU
      ↓
Kernel Network Processing
      ↓
Socket
      ↓
Application

 

When the NIC receives a packet, it can notify the CPU through an interrupt.

The CPU then begins processing the corresponding network event.

This approach has an important advantage:

The CPU does not need to continuously check the NIC when there is no traffic.

Therefore, CPU resources can be used for other work while the network is idle.

This is a good general-purpose design.

However, interrupts also have a cost.

---

3. Why Can Interrupts Add Latency?

An interrupt is not simply:

Packet → Application

There is a processing path between the hardware event and the application.

A simplified representation is:

NIC
 ↓
Interrupt
 ↓
Kernel
 ↓
Network Stack
 ↓
Socket
 ↓
Application

 

The CPU may need to transition through several layers before the application receives the data.

There can also be scheduling and batching effects.

For ordinary server workloads, these costs are usually acceptable.

For extremely latency-sensitive workloads, however, even small and variable delays can become important.

This leads to an important question:

Do we really want the CPU to wait for an interrupt every time?

---

4. Polling

Polling takes a different approach.

Instead of waiting for the NIC to notify the CPU, the CPU repeatedly checks whether data is available.

Conceptually:

while (running)
{
    check_for_packet();

    if (packet_available)
        process_packet();
}

The CPU is continuously active.

The advantage is obvious:

No waiting
   ↓
Check immediately
   ↓
Process packet

 

There is no need to wait for an interrupt to wake the processing path.

But there is also an obvious cost:

The CPU core remains busy even when there is no traffic.

---

5. Interrupts vs. Polling

A simplified comparison looks like this:

Interrupt-DrivenPolling

CPU Usage	Lower when idle	High
Idle Behavior	Wait	Continuously check
Response Path	Event-driven	Continuously active
Latency Predictability	Can vary	Can be more deterministic
Power Consumption	Generally lower	Generally higher
Dedicated Core	Not always required	Often useful
Low-Latency Potential	Good	Very high in suitable designs

Neither approach is universally better.

The important question is:

What is the workload?

---

6. Linux NAPI

Linux networking uses an important mechanism called NAPI (New API).

NAPI combines interrupt-driven notification with polling-based packet processing.

A simplified concept is:

             Packet Arrives
                   ↓
                  NIC
                   ↓
              Interrupt
                   ↓
             NAPI Polling
                   ↓
            Network Stack
                   ↓
                Socket
                   ↓
             Application

 

The basic idea is that an interrupt can notify the kernel that packets have arrived, after which packet processing can switch to polling.

This helps avoid handling a separate interrupt for every packet when traffic becomes heavy.

The important concept is:

Interrupts can be used to detect activity, while polling can be used to process packets efficiently under load.

This hybrid approach is one of the key ideas behind modern Linux networking.

---

7. Why Packet Rate Changes the Optimal Strategy

Consider two different workloads.

Low Traffic

Packet
   ↓
          idle
   ↓
Packet
   ↓
          idle
   ↓
Packet

There may be long periods with no packets.

Polling continuously would consume CPU even during those idle periods.

Interrupt-driven processing can therefore be attractive.

---

High Traffic

Now consider:

Packet Packet Packet Packet Packet Packet
   ↓
   ↓
   ↓
   ↓
   ↓
   ↓
Continuous Traffic

 

 

If the system generates a large number of interrupts, the overhead of handling individual interrupts can become significant.

Polling and batching can become more attractive.

This is one reason modern network processing often uses a hybrid approach rather than relying purely on interrupts.

---

8. Interrupt Moderation and Batching

Network hardware can also use mechanisms that reduce the number of interrupts generated.

Instead of:

Packet 1 → Interrupt
Packet 2 → Interrupt
Packet 3 → Interrupt
Packet 4 → Interrupt

the system may process several packets together:

Packet 1
Packet 2
Packet 3
Packet 4
     ↓
  Batch
     ↓
Processing

This can improve throughput and reduce per-packet overhead.

But there is a trade-off.

If the system waits to accumulate more packets, the first packet may experience additional delay.

Therefore:

Batching improves efficiency, but batching can also add latency.

For high-throughput systems this can be beneficial.

For ultra-low-latency systems, the balance must be measured carefully.

---

9. Latency vs. CPU Efficiency

This brings us back to the fundamental trade-off.

                    CPU Efficiency
                         ↑
                         │
Interrupt-driven        │
                         │
                         │
                         │
                         │
                         └────────────────→ Latency Optimization

                              Polling

This diagram is intentionally simplified.

The real relationship depends on:

• NIC
• Driver
• Kernel version
• CPU architecture
• Packet rate
• Packet size
• Number of queues
• Core assignment
• NUMA topology
• Application architecture

Therefore, there is no universal "fastest networking configuration."

---

10. epoll Is Another Layer

In the previous article, we discussed epoll.

It is important to distinguish NIC packet processing from application-level event notification.

A simplified path is:

NIC
 ↓
Driver / Kernel
 ↓
Socket Buffer
 ↓
epoll
 ↓
Application

 

epoll_wait() waits for file descriptors to become ready for I/O.

When no event is available, the calling thread can block.

When an event becomes ready, epoll_wait() returns the relevant file descriptors.

This makes epoll fundamentally different from continuously executing:

while (1)
{
    check();
}

The latter is a polling loop.

---

11. epoll and Busy Polling Are Not Exactly the Same Thing

This distinction is important.

There are several different levels of polling.

Application-Level Polling

Application
    ↓
read()
    ↓
check
    ↓
check
    ↓
check

Socket / Kernel-Level Busy Polling

The kernel can use polling mechanisms to reduce the time between packet arrival and socket-level processing.

NIC / Driver Polling

The network driver can poll the NIC receive queues.

User-Space Networking

Some architectures move more of the packet-processing path into user space.

Examples include technologies such as:

• DPDK
• User-space networking frameworks
• Kernel-bypass networking

These approaches are much deeper architectural changes than simply replacing epoll_wait() with a busy loop.

---

12. CPU Affinity Becomes Important

Polling introduces another problem:

Which CPU is doing the polling?

Suppose we have:

NIC
 ↓
Polling Thread
 ↓
Strategy
 ↓
Risk
 ↓
Order

If the polling thread moves between CPU cores, cache locality can suffer.

This connects directly to Deep Dive #3:

CPU Affinity and Core Pinning

A low-latency architecture may therefore dedicate a CPU core to a specific processing path.

For example:

Core 0
Market Data Polling

Core 1
Order Book

Core 2
Strategy

Core 3
Risk / Order Management

The actual design depends heavily on workload and hardware topology.

---

13. NUMA Matters Again

We also discussed NUMA in Deep Dive #4.

The network path can therefore be viewed as:

NIC
 ↓
NUMA Node
 ↓
CPU Core
 ↓
Cache
 ↓
Memory
 ↓
Application

If the NIC is attached to NUMA Node 0 but the processing thread is running primarily on NUMA Node 1, network data may cross the NUMA interconnect.

This can introduce additional overhead.

Therefore:

NIC locality is part of CPU locality.

The complete topology should be considered:

NIC
 ↕
CPU
 ↕
Memory

rather than optimizing each component independently.

---

14. Market Data Is a Good Example

Consider a high-volume market data feed.

A simplified architecture might be:

Exchange
   ↓
NIC
   ↓
Packet Receive
   ↓
Decode
   ↓
Order Book Update
   ↓
Strategy
   ↓
Risk Check
   ↓
Order
   ↓
FEP

Every stage can introduce latency.

If the system spends significant time waiting for:

NIC
 ↓
Interrupt
 ↓
Scheduler
 ↓
Application

the resulting latency may become visible in the end-to-end measurement.

This is why low-latency system design often focuses on removing unnecessary waiting.

---

15. But Busy Polling Is Not Free

It is tempting to say:

"If polling is faster, just poll everything."

That is usually not a good design principle.

A polling thread may consume an entire CPU core.

For example:

Core 0
████████████████████
100% Polling

Even when there is no network traffic.

This can be acceptable if Core 0 is intentionally dedicated to a latency-sensitive workload.

But it may be wasteful if the workload is intermittent.

The correct question is:

Is the latency reduction worth the CPU cost?

---

16. Tail Latency Matters

Average latency can hide the behavior of a system.

Consider:

Average latency: 10 μs

That number alone tells us very little.

We should also examine:

P50
P95
P99
P99.9

For example, a system might have a low median latency but occasional long delays caused by:

• Interrupt behavior
• Scheduling
• CPU migration
• Cache misses
• NUMA access
• Lock contention
• Queue buildup

This is why low-latency engineering is often about latency predictability, not simply reducing the average.

---

17. What Should We Measure?

Before changing the networking architecture, measure the system.

Useful measurements include:

Network

• Packet rate
• Packet size
• RX/TX throughput
• Packet drops
• Queue utilization

CPU

• CPU utilization
• CPU migration
• Context switches
• CPU cycles
• IPC

Cache / Memory

• Cache misses
• Memory bandwidth
• NUMA locality
• Remote memory access

Application

• Packet arrival timestamp
• Receive timestamp
• Decode completion
• Strategy completion
• Order submission
• Exchange response

Latency

• P50
• P95
• P99
• P99.9
• Maximum latency

The objective is not simply:

"Make the CPU usage 100%."

The objective is:

"Understand where the latency comes from."

---

18. A Practical Architecture

For a latency-sensitive trading application, a simplified architecture could look like:

             Network
                ↓
               NIC
                ↓
        Interrupt / NAPI
                ↓
         Dedicated CPU
                ↓
        Packet Processing
                ↓
          Shared / Local
             Data
                ↓
            Strategy
                ↓
          Risk Check
                ↓
          Order Manager
                ↓
             FEP
                ↓
           Exchange

The important part is not that every system should use this exact architecture.

The important part is understanding the entire path.

---

19. FontesFintech Engineering Approach

At FontesFintech, we consider low-latency networking as a complete system rather than a single API choice.

The question is not simply:

"Should we use epoll or polling?"

Instead, we ask:

How does the packet arrive?
        ↓
How does the NIC notify the system?
        ↓
How does the kernel process it?
        ↓
How does the application receive it?
        ↓
Which CPU processes it?
        ↓
Where is the data stored?
        ↓
How does the application react?
        ↓
How quickly is the order transmitted?

This leads to a broader optimization chain:

NIC → Interrupt/Polling → Kernel → CPU → Cache → Memory → Application → FEP

Each stage should be measured before it is optimized.

---

Conclusion

Interrupts and polling are not simply competing technologies.

They represent different ways of balancing:

Latency
CPU Usage
Throughput
Power
Predictability

Interrupt-driven processing is efficient when the system spends significant time idle.

Polling can provide a more continuously active processing path and may be useful for extremely latency-sensitive workloads.

Modern Linux networking also combines these ideas through mechanisms such as NAPI.

For trading systems, the most important principle is therefore not:

"Always use polling."

or

"Always use interrupts."

It is:

Choose the waiting and packet-processing strategy based on the actual workload, hardware topology, and measured latency.

And once again:

Measure First. Optimize with Evidence.

---

Next: Trading Engineering — Deep Dive #7

NIC Offload and Kernel Networking in Low-Latency Trading Systems

We will look deeper into the NIC itself — hardware offload, checksum processing, segmentation, receive/transmit queues, RSS, and how the network card interacts with CPU cores in high-performance trading systems.