Paper: arXiv 2502.09172

Authors: Peer Nagy, Sascha Frey, Kang Li, Bidipta Sarkar, Svitlana Vyetrenko, Stefan Zohren, Ani Calinescu, Jakob Foerster

Abstract

While financial data presents one of the most challenging and interesting sequence modelling tasks due to high noise, heavy tails, and strategic interactions, progress in this area has been hindered by the lack of consensus on quantitative evaluation paradigms. To address this, we present LOB-Bench, a benchmark, implemented in python, designed to evaluate the quality and realism of generative message-by-order data for limit order books (LOB) in the LOBSTER format. Our framework measures distributional differences in conditional and unconditional statistics between generated and real LOB data, supporting flexible multivariate statistical evaluation. The benchmark also includes features commonly used LOB statistics such as spread, order book volumes, order imbalance, and message inter-arrival times, along with scores from a trained discriminator network. Lastly, LOB-Bench contains “market impact metrics”, i.e. the cross-correlations and price response functions for specific events in the data. We benchmark generative autoregressive state-space models, a (C)GAN, as well as a parametric LOB model and find that the autoregressive GenAI approach beats traditional model classes.

Complexity vs Empirical Score

  • Math Complexity: 6.5/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper introduces a complex benchmark with statistical and distributional evaluation metrics, showing advanced mathematical formulation, while providing an open-source Python implementation, specific datasets (GOOG, INTC), and backtesting-ready empirical results comparing generative models.

Research Flowchart

  flowchart TD
  A["Research Goal: Develop a benchmark for generative models in finance"] --> B["Data Source: LOBSTER format Limit Order Book data"]
  B --> C["Methodology: LOB-Bench Framework"]
  C --> D["Computational Metrics: Distributional differences, LOB statistics, Market impact"]
  D --> E["Evaluation: C)GAN, Autoregressive models, Parametric models"]
  E --> F["Findings: Autoregressive GenAI outperforms traditional models"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes LOB-Bench, a Python benchmark for evaluating generative message‑by‑order limit order book data.

Source sentence
we present LOB-Bench, a benchmark, implemented in python, designed to evaluate the quality and realism of generative message-by-order data for limit order books (LOB) in the LOBSTER format.

Assumes 2

  • Assumes negligible market impact when using historical data for backtesting. market
    Source sentence
    Traditionally, it is common to use historical market data to train and backtest a trading strategy, thereby making the assumption of negligible market impact.
  • Evaluates models in the autoregressive sampling regime (generating one token at a time). model
    Source sentence
    we want to evaluate the pre-trained models in the sampling regime to unlock the mentioned use-cases.

Data

  • LOBSTER · LOBSTER order-book public
    Source sentence
    generative message-by-order data for limit order books (LOB) in the LOBSTER format.
  • GOOG 2023 order-book unknown
    2023 · GOOG
    Source sentence
    trained and tested on GOOG data from 2023, achieves a Receiver Operating Characteristic (ROC) score of 0.83

Recognised sources: LOBSTER

Costs

Not applicable The paper reports no trading or execution result.

Finds 4

  • Autoregressive generative AI models outperform traditional parametric and GAN baselines on the LOB-Bench evaluation. performance
    Source sentence
    we benchmark generative autoregressive state-space models, a (C)GAN, as well as a parametric LOB model and find that the autoregressive GenAI approach beats traditional model classes.
  • The LOBS5 model reproduces most features of expected impact response functions, unlike the baseline. statistical
    Source sentence
    In contrast to the baseline, LOBS5 reproduces most features of the expected impact response functions.
  • The discriminator achieves an ROC score of 0.83, indicating generated data can be reasonably distinguished. performance
    Source sentence
    trained and tested on GOOG data from 2023, achieves a Receiver Operating Characteristic (ROC) score of 0.83
  • Current generative models do not improve mid‑price trend prediction when mixed with real data. negative
    Source sentence
    the results show that current generative models are not yet able to generate data aiding this mid-price trend prediction task.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 28 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 6 papers in the archive (5 by independent authors): Prospects of Imitating Trading Agents in the Stock Market (extends it)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.