Paper: arXiv 2308.01915
Authors: Matteo Prata, Giuseppe Masi, Leonardo Berti, Viviana Arrigoni, Andrea Coletta, Irene Cannistraci, Svitlana Vyetrenko, Paola Velardi, Novella Bartolini
Abstract
The recent advancements in Deep Learning (DL) research have notably influenced the finance sector. We examine the robustness and generalizability of fifteen state-of-the-art DL models focusing on Stock Price Trend Prediction (SPTP) based on Limit Order Book (LOB) data. To carry out this study, we developed LOBCAST, an open-source framework that incorporates data preprocessing, DL model training, evaluation and profit analysis. Our extensive experiments reveal that all models exhibit a significant performance drop when exposed to new data, thereby raising questions about their real-world market applicability. Our work serves as a benchmark, illuminating the potential and the limitations of current approaches and providing insight for innovative solutions.
Complexity vs Empirical Score
- Math Complexity: 6.5/10
- Empirical Rigor: 7.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper involves advanced deep learning architectures and statistical evaluation (high math), and features a comprehensive benchmarking study with an open-source framework, multiple datasets, and profit analysis (high empirical rigor).
Research Flowchart
flowchart TD
A["Research Goal:<br>Assess DL Model Robustness &<br>Generalizability for LOB-based SPTP"] --> B["Methodology:<br>Develop LOBCAST Framework"]
subgraph B ["Methodology: LOBCAST Framework"]
B1["Data Preprocessing"] --> B2["DL Model Training"] --> B3["Evaluation &<br>Profit Analysis"]
end
C["Inputs:<br>Limit Order Book Data<br>Including Non-Stationary Regimes"] --> B
B --> D{"Computational Process:<br>Train on Known vs.<br>Test on New Data Regimes"}
D --> E["Key Findings:<br>1. Significant Performance Drop<br>on New Data<br>2. Limited Real-World<br>Market Applicability<br>3. Benchmark for Future<br>Innovative Solutions"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper releases LOBCAST, an open‑source modular framework for preprocessing, training, testing and profit analysis of LOB‑based stock price trend models.
Source sentence
We release a highly modular open-source framework called LOBCAST1, to pre-process data, train, and test stock market models.
Assumes 6
- Each trade uses a fixed size of one share, implying negligible market impact. execution
Source sentence
we set the number of shares per trade to a fixed value of 1, simplifying our analysis and assuming a negligible market impact.
- The simulated trader starts with an initial capital of $10,000. execution
Source sentence
our simulated trader begins with an initial capital of $10.000
- No transaction fees are applied in the trading simulation. execution
Source sentence
we make the assumption of no transaction fees.
- Trading signals are generated every 10 order‑book events. model
Source sentence
The trading strategy relies on the models and operates by generating signals every 10 events to predict subsequent price movements.
Show 2 more
- Orders are filled at the next open price, effectively assuming no slippage. execution
Source sentence
The orders are filled at the next open price.
- Trades are executed at the mid‑price, ignoring market impact and other frictions. execution
Source sentence
we assume the trades execution at the mid-price, and we ignore transaction costs
Data
- FI-2010 order-book public
Source sentence
The most widely spread public LOB dataset is FI-2010 which is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0) and was proposed in 2017 by Ntakaris et al.
- LOB-2021 order-book commercial
Source sentence
datasets called LOB-2021 and LOB-2022, extrapolated from the LOBSTER dataset
- LOB-2022 order-book commercial
Source sentence
datasets called LOB-2021 and LOB-2022, extrapolated from the LOBSTER dataset
- LOBSTER order-book commercial
Source sentence
To test the generalizability of the models in a more realistic scenario, we used data extracted from LOBSTER [13], an online LOB data provider for order book data, which is not available for free
Costs
Unclear - sources disagree
Source sentence (the authors' words)
we assume the trades execution at the mid-price, and we ignore transaction costs
Finds 4
- All fifteen DL models suffer a significant performance drop when evaluated on unseen LOBSTER data, questioning real‑world applicability. negative
Source sentence
Our extensive experiments reveal that all models exhibit a significant performance drop when exposed to new data, thereby raising questions about their real-world market applicability.
- The BINCTABL model achieves the highest average F1‑score (82.6% ± 7.0) and the best robustness score (99.7) on the FI‑2010 dataset. performance
Source sentence
... the BINCTABL model demonstrates the highest F1-score when averaged over the seeds and prediction horizons, achieving an average of 82.6% ± 7.0.
- Generalizability tests show overall F1‑scores drop to 48‑61% on LOB‑2021/2022, far below FI‑2010 results. performance
Source sentence
the overall performance of all models on the LOB-2021/2022 dataset is still significantly lower than on the FI-2010 dataset, ranging 48-61% in F1-score.
- Trading simulation profits are positive only for stocks with high daily returns (e.g., LSTR, NFLX) and negative for stocks with high negative returns (e.g., SOFI, SHLS). performance
Source sentence
the two stocks with the highest positive daily returns (namely LSTR and NFLX) are the only ones for which the strategy is profitable. On the other hand, the two stocks with the highest negative daily returns (SOFI and SHLS) are the ones for which most models show a negative return.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 18 related papers). Coverage is partial, so this is neither replication nor refutation.
8 papers in the archive : Explainable Patterns in Cryptocurrency Microstructure ; An Efficient deep learning model to Predict Stock Price … ; HLOB -- Information Persistence and Structure in Limit Order …
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.