Paper: arXiv 2502.09172
Authors: Peer Nagy, Sascha Frey, Kang Li, Bidipta Sarkar, Svitlana Vyetrenko, Stefan Zohren, Ani Calinescu, Jakob Foerster
Abstract
While financial data presents one of the most challenging and interesting sequence modelling tasks due to high noise, heavy tails, and strategic interactions, progress in this area has been hindered by the lack of consensus on quantitative evaluation paradigms. To address this, we present LOB-Bench, a benchmark, implemented in python, designed to evaluate the quality and realism of generative message-by-order data for limit order books (LOB) in the LOBSTER format. Our framework measures distributional differences in conditional and unconditional statistics between generated and real LOB data, supporting flexible multivariate statistical evaluation. The benchmark also includes features commonly used LOB statistics such as spread, order book volumes, order imbalance, and message inter-arrival times, along with scores from a trained discriminator network. Lastly, LOB-Bench contains “market impact metrics”, i.e. the cross-correlations and price response functions for specific events in the data. We benchmark generative autoregressive state-space models, a (C)GAN, as well as a parametric LOB model and find that the autoregressive GenAI approach beats traditional model classes.
Complexity vs Empirical Score
- Math Complexity: 6.5/10
- Empirical Rigor: 7.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper introduces a complex benchmark with statistical and distributional evaluation metrics, showing advanced mathematical formulation, while providing an open-source Python implementation, specific datasets (GOOG, INTC), and backtesting-ready empirical results comparing generative models.
Research Flowchart
flowchart TD A["Research Goal: Develop a benchmark for generative models in finance"] --> B["Data Source: LOBSTER format Limit Order Book data"] B --> C["Methodology: LOB-Bench Framework"] C --> D["Computational Metrics: Distributional differences, LOB statistics, Market impact"] D --> E["Evaluation: C)GAN, Autoregressive models, Parametric models"] E --> F["Findings: Autoregressive GenAI outperforms traditional models"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes LOB-Bench, a Python benchmark for evaluating generative message‑by‑order limit order book data.
Source sentence
we present LOB-Bench, a benchmark, implemented in python, designed to evaluate the quality and realism of generative message-by-order data for limit order books (LOB) in the LOBSTER format.
Assumes 2
- Assumes negligible market impact when using historical data for backtesting. market
Source sentence
Traditionally, it is common to use historical market data to train and backtest a trading strategy, thereby making the assumption of negligible market impact.
- Evaluates models in the autoregressive sampling regime (generating one token at a time). model
Source sentence
we want to evaluate the pre-trained models in the sampling regime to unlock the mentioned use-cases.
Data
Costs
Not applicable
Finds 4
- Autoregressive generative AI models outperform traditional parametric and GAN baselines on the LOB-Bench evaluation. performance
Source sentence
we benchmark generative autoregressive state-space models, a (C)GAN, as well as a parametric LOB model and find that the autoregressive GenAI approach beats traditional model classes.
- The LOBS5 model reproduces most features of expected impact response functions, unlike the baseline. statistical
Source sentence
In contrast to the baseline, LOBS5 reproduces most features of the expected impact response functions.
- The discriminator achieves an ROC score of 0.83, indicating generated data can be reasonably distinguished. performance
Source sentence
trained and tested on GOOG data from 2023, achieves a Receiver Operating Characteristic (ROC) score of 0.83
- Current generative models do not improve mid‑price trend prediction when mixed with real data. negative
Source sentence
the results show that current generative models are not yet able to generate data aiding this mid-price trend prediction task.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 28 related papers). Coverage is partial, so this is neither replication nor refutation.
6 papers in the archive : Prospects of Imitating Trading Agents in the Stock Market
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.