Paper: arXiv 2511.00190
Authors: Andrea Macrì, Sebastian Jaimungal, Fabrizio Lillo
Abstract
Reinforcement Learning (RL) applied to financial problems has been the subject of a lively area of research. The use of RL for optimal trading strategies that exploit latent information in the market is, to the best of our knowledge, not widely tackled. In this paper we study an optimal trading problem, where a trading signal follows an Ornstein-Uhlenbeck process with regime-switching dynamics. We employ a blend of RL and Recurrent Neural Networks (RNN) in order to make the most at extracting underlying information from the trading signal with latent parameters. The latent parameters driving mean reversion, speed, and volatility are filtered from observations of the signal, and trading strategies are derived via RL. To address this problem, we propose three Deep Deterministic Policy Gradient (DDPG)-based algorithms that integrate Gated Recurrent Unit (GRU) networks to capture temporal dependencies in the signal. The first, a one -step approach (hid-DDPG), directly encodes hidden states from the GRU into the RL trader. The second and third are two-step methods: one (prob-DDPG) makes use of posterior regime probability estimates, while the other (reg-DDPG) relies on forecasts of the next signal value. Through extensive simulations with increasingly complex Markovian regime dynamics for the trading signal’s parameters, as well as an empirical application to equity pair trading, we find that prob-DDPG achieves superior cumulative rewards and exhibits more interpretable strategies. By contrast, reg-DDPG provides limited benefits, while hid-DDPG offers intermediate performance with less interpretable strategies. Our results show that the quality and structure of the information supplied to the agent are crucial: embedding probabilistic insights into latent regimes substantially improves both profitability and robustness of reinforcement learning-based trading strategies.
Complexity vs Empirical Score
- Math Complexity: 7.5/10
- Empirical Rigor: 6.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper involves advanced stochastic calculus (Ornstein-Uhlenbeck SDEs, regime-switching Markov chains) and complex RL algorithms (DDPG with GRUs), indicating high math density. It is grounded in empirical testing via simulations and real equity pair trading data, though the backtest-ready implementation details are not fully specified in the summary.
Research Flowchart
flowchart TD
A["Research Goal: Optimal Trading<br>with Partial/Latent Info"] --> B["Data & Environment"]
B --> C{"RL Agent Architecture"}
subgraph B ["Inputs"]
B1["Ornstein-Uhlenbeck Signal<br>with Regime-Switching"]
B2["Market Observations"]
end
subgraph C ["Core Methodology: DDPG + GRU"]
direction TB
C1["GRU: Extract Temporal<br>Latent Features"]
C2["RL Actor-Critic<br>DDPG Policy Optimization"]
C1 --> C2
end
C --> D{"Algorithm Variants"}
D --> E["hid-DDPG<br>Direct Hidden States"]
D --> F["prob-DDPG<br>Posterior Regime<br>Probabilities"]
D --> G["reg-DDPG<br>Signal Forecast"]
E & F & G --> H["Simulation &<br>Equity Pair Trading"]
H --> I{"Key Findings / Outcomes"}
I --> J["prob-DDPG: Superior<br>Profit & Interpretability"]
I --> K["reg-DDPG: Limited Benefit"]
I --> L["hid-DDPG: Intermediate<br>Performance"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes three Deep Deterministic Policy Gradient (DDPG) algorithms that integrate Gated Recurrent Unit (GRU) networks to solve an optimal trading problem with a mean‑reverting Ornstein‑Uhlenbeck signal and latent regime information.
Source sentence
To address this problem, we propose three Deep Deterministic Policy Gradient (DDPG)–based algorithms that integrate Gated Recurrent Unit (GRU) networks to capture temporal dependencies in the signal.
Assumes 6
- The trading signal follows an Ornstein‑Uhlenbeck process with regime‑switching dynamics governed by independent Markov chains. model
Source sentence
we study an optimal trading problem, where a trading signal follows an Ornstein–Uhlenbeck process with regime‑switching dynamics.
- Transaction costs are a linear per‑unit fee λ ≥ 0 applied to each trade volume. execution
Source sentence
Whenever the trader performs a trade of volume q_t, she pays a transaction cost per unit volume of λ ≥0.
- The agent can hold inventory within a bounded interval [‑I_max, I_max] and starts with zero inventory. execution
Source sentence
the agent can hold inventories within the interval I ∈[‑Imax, Imax] and we always start the testing episodes of trading with S0 = 1 and I0 = 0.
- The trader is risk‑neutral and maximises the expected discounted sum of rewards. model
Source sentence
a risk neutral trader who maximises the expected discounted profits from trading.
Show 2 more
- The latent parameters (mean reversion, speed, volatility) are filtered from observations of the signal. data
Source sentence
The latent parameters driving mean reversion, speed, and volatility are filtered from observations of the signal.
- The pair‑trading assets are co‑integrated, allowing a stationary linear combination to be used as the trading portfolio. model
Source sentence
We see that both tests indicate strong evidence of at least one co‑integrating vector
Data
- Intel (INTC) and Merrill Lynch Semiconductor ETF (SMH) high‑frequency limit order book dat order-book commercial
Source sentence
We consider two assets traded on the NASDAQ between the 29th of August 2025 and the 5th of September 2025, the Intel (INTC) stock and the Merril Lynch semi‑conductors ETF (SMH)
Costs
Modeled
A per‑unit transaction cost λ ≥ 0 is subtracted from the reward for each trade volume.
Source sentence
Whenever the trader performs a trade of volume q_t, she pays a transaction cost per unit volume of λ ≥0. As a result, the reward r_t from the trading action at time t is r_t(q_t, S_t, S_{t+1}, λ) := I_{t+1} (S_{t+1}‑S_t) ‑λ |q_t| .
Finds 5
- prob‑DDPG consistently achieves the highest cumulative trading rewards and the most interpretable strategies in both synthetic and real‑world pair‑trading experiments. performance
Source sentence
we find that prob‑DDPG achieves superior cumulative rewards and exhibits more interpretable strategies.
- reg‑DDPG provides limited benefits, offering little improvement over baseline. performance
Source sentence
By contrast, reg‑DDPG provides limited benefits.
- hid‑DDPG yields intermediate performance with less interpretable strategies. performance
Source sentence
hid‑DDPG offers intermediate performance with less interpretable strategies.
- Providing posterior probability estimates of the mean‑reversion regimes leads to the highest trading rewards, confirming the importance of structured information. performance
Source sentence
The numerical results show that the agent, when equipped with posterior probability estimates of the mean reversion regimes (which it learns in a first stage), achieves the highest trading rewards.
- The conclusions about the superiority of prob‑DDPG hold when applied to real market pair‑trading data. performance
Source sentence
Finally, we find that these conclusions remain valid when the agent faces trading problems with real market data, when dealing with a pair trading problem for two co‑integrated assets.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 27 related papers). Coverage is partial, so this is neither replication nor refutation.
1 paper in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (37)
- On the Optimal Selection of Time-Lag Embedding Dimension for Deep Learning Approaches in Financial Forecasting With Big Data (2026)
- Development of a cryptocurrency price prediction model: leveraging GRU and LSTM for Bitcoin, Litecoin and Ethereum (2025)
- SA cooperative deep learning model for stock market prediction using deep autoencoder and sentiment 451 analysis (2022)
- Reinforcement learning for algorithmic trading (2023)
- Deep Q-learning for Nash equilibria: Nash-DQN (2022)
- Double deep q-learning for optimal execution (2021)
- Trading algorithms with learning in latent alpha models (2019)
- Incorporating order-flow into optimal execution (2016)
and 29 more
- Dynamic trading with predictable returns and transaction costs (2013)
- Continuous control with deep reinforcement learning (2015) — link
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation (2014)
- Algorithmic and high-frequency trading (2015)
- Deep Reinforcement Learning for Active High Frequency Trading (2023) — link
- Dynamic portfolio optimization with inverse covariance clustering (2023) — link
- Stock Market Prediction via Deep Learning Techniques: A Survey — → 2212.12717
- Using deep learning for price prediction by exploiting stationary limit order book features (2020)
- Universal features of price formation in financial markets: perspectives from deep learning (2021)
- Deeplob: Deep convolutional neural networks for limit order books (2019)
- Deep order flow imbalance: Extracting alpha at multiple horizons from the limit order book (2023)
- Deep reinforcement learning for trading (2019) — link
- Deep reinforcement learning for automated stock trading: An ensemble strategy (2020)
- Recent advances in reinforcement learning in finance (2023)
- Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014) — link
- Algorithmic trading: winning strategies and their rationale (2013)
- A new approach to the economic analysis of nonstationary time series and the business cycle (1989) — link
- Planning and acting in partially observable stochastic domains (1998)
- Optimal control of Markov processes with incomplete state information I (1965)
- Multi-agent reinforcement learning: Foundations and modern approaches (2024)
- Some applications of stochastic differential equations to optimal nonlinear filtering (1964)
- Q-learning for stochastic control under general information structures and non-Markovian environments (2023) — link
- Statistical analysis of cointegration vectors (1988)
- Estimation and hypothesis testing of cointegration vectors in Gaussian vector autoregressive models (1991)
- The optimal control of partially observable Markov processes over a finite horizon (1973)
- Markov chains (1998)
- Recurrent Neural Networks and Universal Approximation of Bayesian Filters (2023)
- Recurrent Networks, Hidden States and Beliefs in Partially Observable Environments (2022)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014) — link
1 of 37 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.