Paper: arXiv 2308.01915

Authors: Matteo Prata, Giuseppe Masi, Leonardo Berti, Viviana Arrigoni, Andrea Coletta, Irene Cannistraci, Svitlana Vyetrenko, Paola Velardi, Novella Bartolini

Abstract

The recent advancements in Deep Learning (DL) research have notably influenced the finance sector. We examine the robustness and generalizability of fifteen state-of-the-art DL models focusing on Stock Price Trend Prediction (SPTP) based on Limit Order Book (LOB) data. To carry out this study, we developed LOBCAST, an open-source framework that incorporates data preprocessing, DL model training, evaluation and profit analysis. Our extensive experiments reveal that all models exhibit a significant performance drop when exposed to new data, thereby raising questions about their real-world market applicability. Our work serves as a benchmark, illuminating the potential and the limitations of current approaches and providing insight for innovative solutions.

Complexity vs Empirical Score

  • Math Complexity: 6.5/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper involves advanced deep learning architectures and statistical evaluation (high math), and features a comprehensive benchmarking study with an open-source framework, multiple datasets, and profit analysis (high empirical rigor).

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Assess DL Model Robustness &<br>Generalizability for LOB-based SPTP"] --> B["Methodology:<br>Develop LOBCAST Framework"]

  subgraph B ["Methodology: LOBCAST Framework"]
      B1["Data Preprocessing"] --> B2["DL Model Training"] --> B3["Evaluation &<br>Profit Analysis"]
  end

  C["Inputs:<br>Limit Order Book Data<br>Including Non-Stationary Regimes"] --> B

  B --> D{"Computational Process:<br>Train on Known vs.<br>Test on New Data Regimes"}

  D --> E["Key Findings:<br>1. Significant Performance Drop<br>on New Data<br>2. Limited Real-World<br>Market Applicability<br>3. Benchmark for Future<br>Innovative Solutions"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper releases LOBCAST, an open‑source modular framework for preprocessing, training, testing and profit analysis of LOB‑based stock price trend models.

Source sentence
We release a highly modular open-source framework called LOBCAST1, to pre-process data, train, and test stock market models.

Assumes 6

  • Each trade uses a fixed size of one share, implying negligible market impact. execution
    Source sentence
    we set the number of shares per trade to a fixed value of 1, simplifying our analysis and assuming a negligible market impact.
  • The simulated trader starts with an initial capital of $10,000. execution
    Source sentence
    our simulated trader begins with an initial capital of $10.000
  • No transaction fees are applied in the trading simulation. execution
    Source sentence
    we make the assumption of no transaction fees.
  • Trading signals are generated every 10 order‑book events. model
    Source sentence
    The trading strategy relies on the models and operates by generating signals every 10 events to predict subsequent price movements.
Show 2 more
  • Orders are filled at the next open price, effectively assuming no slippage. execution
    Source sentence
    The orders are filled at the next open price.
  • Trades are executed at the mid‑price, ignoring market impact and other frictions. execution
    Source sentence
    we assume the trades execution at the mid-price, and we ignore transaction costs

Data

  • FI-2010 · Ntakaris et al. order-book public
    five Finnish companies (Kesko Oyj, Outokumpu Oyj, Sampo, Rautaruukki, Wärtsilä)
    Source sentence
    The most widely spread public LOB dataset is FI-2010 which is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0) and was proposed in 2017 by Ntakaris et al.
  • LOB-2021 · LOBSTER (reconstructed NASDAQ data) order-book commercial
    Source sentence
    datasets called LOB-2021 and LOB-2022, extrapolated from the LOBSTER dataset
  • LOB-2022 · LOBSTER (reconstructed NASDAQ data) order-book commercial
    Source sentence
    datasets called LOB-2021 and LOB-2022, extrapolated from the LOBSTER dataset
  • LOBSTER · LOBSTER (online LOB data provider) order-book commercial
    Source sentence
    To test the generalizability of the models in a more realistic scenario, we used data extracted from LOBSTER [13], an online LOB data provider for order book data, which is not available for free

Recognised sources: LOBSTER

Costs

Unclear - sources disagree The platform classifier and the paper-text extraction give different answers, so no value is asserted.

Platform classifier: Not applicable

Paper text extraction: Not modeled

Source sentence (the authors' words)
we assume the trades execution at the mid-price, and we ignore transaction costs

Finds 4

  • All fifteen DL models suffer a significant performance drop when evaluated on unseen LOBSTER data, questioning real‑world applicability. negative
    Source sentence
    Our extensive experiments reveal that all models exhibit a significant performance drop when exposed to new data, thereby raising questions about their real-world market applicability.
  • The BINCTABL model achieves the highest average F1‑score (82.6% ± 7.0) and the best robustness score (99.7) on the FI‑2010 dataset. performance
    Source sentence
    ... the BINCTABL model demonstrates the highest F1-score when averaged over the seeds and prediction horizons, achieving an average of 82.6% ± 7.0.
  • Generalizability tests show overall F1‑scores drop to 48‑61% on LOB‑2021/2022, far below FI‑2010 results. performance
    Source sentence
    the overall performance of all models on the LOB-2021/2022 dataset is still significantly lower than on the FI-2010 dataset, ranging 48-61% in F1-score.
  • Trading simulation profits are positive only for stocks with high daily returns (e.g., LSTR, NFLX) and negative for stocks with high negative returns (e.g., SOFI, SHLS). performance
    Source sentence
    the two stocks with the highest positive daily returns (namely LSTR and NFLX) are the only ones for which the strategy is profitable. On the other hand, the two stocks with the highest negative daily returns (SOFI and SHLS) are the ones for which most models show a negative return.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 18 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 8 papers in the archive (6 by independent authors): Explainable Patterns in Cryptocurrency Microstructure (discusses its findings); An Efficient deep learning model to Predict Stock Price … (discusses its findings); HLOB -- Information Persistence and Structure in Limit Order … (uses its method or data)

Code

https://github.com/matteoprata/LOBCAST — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.