Paper: arXiv 2609.32848

Authors: Vincent Maciejewski

Abstract

HFT systems are conventionally built as a single-threaded event loop, on the rule that every thread hop adds latency. We test that rule against a measurement study of more than a year of CME market data for the NQ front-month contract, following every packet and matching-engine transaction through the feed’s two exchange timestamps, and checking the results against a live production receiver. Packets arrive in near-critical self-exciting clusters that belong to the matching engine’s transactions, not to how the exchange packs them. The engine often processes consecutive transactions within a fraction of a microsecond, while the market-data publisher sends at most one packet per publisher period of about 7.5 microseconds, so a burst reaches the receiver as a train of packets one period apart. This yields design principles for HFT systems. First, a receiver that handles each packet within one publisher period never queues on arrivals, however bursty the market; there one thread is best. Second, above that period a queueing tail appears, driven by the timing of transactions, not by packet rate or size, and two threads can be better than one: splitting the servicing chain into two stages on separate threads removes most of the tail at the cost of one hop on the median. Third, only the slowest stage matters, so a split pays only if it shortens it. Fourth, just under the period, where the production receiver runs, the remaining tail comes from multi-message packets and variable service times, and the levers are cost per message and spread of service, not thread count. An analytic framework, a burst-limit throughput identity and an exact reduction of the tandem to a single bottleneck server, supports these results.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 9.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: This paper presents a highly rigorous empirical study of HFT system design principles, backed by a sophisticated analytical framework. Its novelty lies in challenging conventional wisdom regarding single-threaded event loops through detailed measurement and simulation. The comprehensive analysis and clear recommendations make it a significant contribution.

Research Flowchart

  flowchart TD
    A["Research Goal: Validate HFT Event Loop Rule vs. Latency"] --> B{"Key Methodology: Measurement Study of CME Market Data"};
    B --> C["Data/Inputs: >1 Year NQ Front-Month CME Market Data, Live Production Receiver Logs"];
    C --> D{"Computational Process: Packet/Transaction Tracking, Latency Analysis, Queueing Models"};
    D --> E["Key Finding 1: Packets arrive in near-critical self-exciting clusters belonging to matching engine transactions"];
    E --> F["Key Finding 2: HFT Design Principles based on Publisher Period & Transaction Timing"];
    F --> G["Key Outcome: Analytic Framework, Burst-Limit Throughput Identity, Tandem Reduction for HFT System Design"];

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes design principles for HFT receivers that depend on the relationship between packet arrival bursts and the publisher period, recommending a single thread when handling each packet within one publisher period and a two‑stage split when service times exceed that period.

Source sentence
This yields design principles for HFT systems. First, a receiver that handles each packet within one publisher period never queues on arrivals, however bursty the market; there one thread is best.

Assumes 6

  • Every thread hop adds latency, motivating a single‑threaded event loop as the baseline design. execution
    Source sentence
    HFT systems are conventionally built as a single-threaded event loop, on the rule that every thread hop adds latency.
  • Packet arrivals form near‑critical self‑exciting clusters that correspond to matching‑engine transactions. market
    Source sentence
    Packets arrive in near-critical self-exciting clusters that belong to the matching engine's transactions, not to how the exchange packs them.
  • The matching engine can process consecutive transactions within a fraction of a microsecond, faster than the publisher’s ~7.5 µs packet period. market
    Source sentence
    The engine often processes consecutive transactions within a fraction of a microsecond, while the market-data publisher sends at most one packet per publisher period of about 7.5 microseconds, so a burst reaches the receiver as a train of packets one period apart.
  • Service time per packet is the dominant factor for queueing; if a receiver processes a packet faster than the publisher period it never queues on arrivals. model
    Source sentence
    First, a receiver that handles each packet within one publisher period never queues on arrivals, however bursty the market; there one thread is best.
Show 2 more
  • When service time exceeds the publisher period, a queueing tail appears that is driven by transaction timing rather than packet rate or size. model
    Source sentence
    Second, above that period a queueing tail appears, driven by the timing of transactions, not by packet rate or size, and two threads can be better than one: splitting the servicing chain into two stages on separate threads removes most of the tail at the cost of one hop on the median.
  • Only the slowest stage in a multi‑stage processing chain determines tail reduction; splitting stages only helps if it shortens that slowest stage. model
    Source sentence
    Third, only the slowest stage matters, so a split pays only if it shortens it.

Data

  • CME market data for the NQ front‑month contract (one year of packets and matching‑engine t · CME market-prices proprietary
    one year · NQ front‑month
    Source sentence
    We test that rule against a measurement study of more than a year of CME market data for the NQ front-month contract, following every packet and matching-engine transaction through the feed's two exchange timestamps, and checking the results against a live production receiver. 13 Conclusion This paper measured a real exchange feed, over a year of NQ packets and matching-engine transactions, from the matching engine

Costs

Not applicable The paper reports no trading or execution result.

Finds 5

  • A receiver that handles each packet within one publisher period never queues on arrivals, regardless of burstiness; a single thread is optimal in this regime. theoretical
    Source sentence
    First, a receiver that handles each packet within one publisher period never queues on arrivals, however bursty the market; there one thread is best.
  • When service times exceed the publisher period, a queueing tail appears, and splitting the processing chain into two threads removes most of the tail at the cost of one hop on the median. theoretical
    Source sentence
    Second, above that period a queueing tail appears, driven by the timing of transactions, not by packet rate or size, and two threads can be better than one: splitting the servicing chain into two stages on separate threads removes most of the tail at the cost of one hop on the median.
  • Only the slowest stage matters for tail reduction; a split only pays off if it shortens that stage. theoretical
    Source sentence
    Third, only the slowest stage matters, so a split pays only if it shortens it.
  • For receivers already operating just under the publisher period, the remaining tail is caused by multi‑message packets and variable service times, not by thread count. theoretical
    Source sentence
    Fourth, just under the period, where the production receiver runs, the remaining tail comes from multi-message packets and variable service times, and the levers are cost per message and spread of service, not thread count. The first is long packets.
  • The matching engine processes transactions sub‑microsecond apart, but the publisher can only emit packets at ~7.5 µs intervals, creating burst trains that drive the receiver’s tail. other
    Source sentence
    The engine often processes consecutive transactions within a fraction of a microsecond, while the market-data publisher sends at most one packet per publisher period of about 7.5 microseconds, so a burst reaches the receiver as a train of packets one period apart. The matching engine often processes one transaction less than a microsecond after the previous one (Table 13).

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 14 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

https://github.com/vincent212/kaspar-hft — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.