{"paper_id":"2502.09172","title":"LOB-Bench: Benchmarking Generative AI for Finance -- an Application to Limit Order Book Data","authors":["Peer Nagy","Sascha Frey","Kang Li","Bidipta Sarkar","Svitlana Vyetrenko","Stefan Zohren","Ani Calinescu","Jakob Foerster"],"paper_date":"2025-02-13","paper_type":"dataset-benchmark","paper_url":"https://arxiv.org/pdf/2502.09172v2","page_url":"https://thequant.space/flowcharts/lob-bench-benchmarking-generative-ai-for-finance----an-appl/","scores":{"empirical_rigor":7,"math_complexity":6.5,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes LOB-Bench, a Python benchmark for evaluating generative message‑by‑order limit order book data.","quote":"we present LOB-Bench, a benchmark, implemented in python, designed to evaluate the quality and realism of generative message-by-order data for limit order books (LOB) in the LOBSTER format.","summary":"While financial data presents one of the most challenging and interesting sequence modelling tasks due to high noise, heavy tails, and strategic interactions, progress in this area has been hindered by the lack of consensus on quantitative evaluation paradigms. To address this, we present…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"Assumes negligible market impact when using historical data for backtesting.","type":"market","stated":"explicit","quote":"Traditionally, it is common to use historical market data to train and backtest a trading strategy, thereby making the assumption of negligible market impact."},{"assumption":"Evaluates models in the autoregressive sampling regime (generating one token at a time).","type":"model","stated":"explicit","quote":"we want to evaluate the pre-trained models in the sampling regime to unlock the mentioned use-cases."}]},"datasets":{"status":"available","items":[{"name":"LOBSTER","provider":"LOBSTER","kind":"order-book","access":"public","period":null,"universe":null,"quote":"generative message-by-order data for limit order books (LOB) in the LOBSTER format.","known_id":"lobster"},{"name":"GOOG 2023","provider":null,"kind":"order-book","access":"unknown","period":"2023","universe":"GOOG","quote":"trained and tested on GOOG data from 2023, achieves a Receiver Operating Characteristic (ROC) score of 0.83"}],"known":[{"id":"lobster","name":"LOBSTER","provider":"LOBSTER","kind":"order-book","access":"commercial","quote":"Using the convention in LOBSTER data, we define the direc-tion (dir) as 1 for events on the bid side and −1 on the ask side.","mentions":6,"used_in":1}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"l2-book"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"unclear","findings":[{"finding":"Autoregressive generative AI models outperform traditional parametric and GAN baselines on the LOB-Bench evaluation.","kind":"performance","quote":"we benchmark generative autoregressive state-space models, a (C)GAN, as well as a parametric LOB model and find that the autoregressive GenAI approach beats traditional model classes."},{"finding":"The LOBS5 model reproduces most features of expected impact response functions, unlike the baseline.","kind":"statistical","quote":"In contrast to the baseline, LOBS5 reproduces most features of the expected impact response functions."},{"finding":"The discriminator achieves an ROC score of 0.83, indicating generated data can be reasonably distinguished.","kind":"performance","quote":"trained and tested on GOOG data from 2023, achieves a Receiver Operating Characteristic (ROC) score of 0.83"},{"finding":"Current generative models do not improve mid‑price trend prediction when mixed with real data.","kind":"negative","quote":"the results show that current generative models are not yet able to generate data aiding this mid-price trend prediction task."}],"reported_performance":null,"out_of_sample":{"method":"holdout","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[],"contradicts":[],"qualifies":[],"compared_with":28,"citations":{"cited_by_in_corpus":6,"by_independent_authors":5,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":1,"discusses_findings":0,"mentions":3,"unclassified":2},"independent_papers":[{"paper":{"paper_id":"2509.00982","title":"Prospects of Imitating Trading Agents in the Stock Market","paper_date":"2025-08-31","authors":["Mateusz Wilinski","Juho Kanniainen"],"empirical_rigor":2.5,"page_url":"https://thequant.space/flowcharts/prospects-of-imitating-trading-agents-in-the-stock-market/"},"how":"extends","quote":null}],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:45:33Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}