Paper: arXiv 2511.00190

Authors: Andrea Macrì, Sebastian Jaimungal, Fabrizio Lillo

Abstract

Reinforcement Learning (RL) applied to financial problems has been the subject of a lively area of research. The use of RL for optimal trading strategies that exploit latent information in the market is, to the best of our knowledge, not widely tackled. In this paper we study an optimal trading problem, where a trading signal follows an Ornstein-Uhlenbeck process with regime-switching dynamics. We employ a blend of RL and Recurrent Neural Networks (RNN) in order to make the most at extracting underlying information from the trading signal with latent parameters. The latent parameters driving mean reversion, speed, and volatility are filtered from observations of the signal, and trading strategies are derived via RL. To address this problem, we propose three Deep Deterministic Policy Gradient (DDPG)-based algorithms that integrate Gated Recurrent Unit (GRU) networks to capture temporal dependencies in the signal. The first, a one -step approach (hid-DDPG), directly encodes hidden states from the GRU into the RL trader. The second and third are two-step methods: one (prob-DDPG) makes use of posterior regime probability estimates, while the other (reg-DDPG) relies on forecasts of the next signal value. Through extensive simulations with increasingly complex Markovian regime dynamics for the trading signal’s parameters, as well as an empirical application to equity pair trading, we find that prob-DDPG achieves superior cumulative rewards and exhibits more interpretable strategies. By contrast, reg-DDPG provides limited benefits, while hid-DDPG offers intermediate performance with less interpretable strategies. Our results show that the quality and structure of the information supplied to the agent are crucial: embedding probabilistic insights into latent regimes substantially improves both profitability and robustness of reinforcement learning-based trading strategies.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper involves advanced stochastic calculus (Ornstein-Uhlenbeck SDEs, regime-switching Markov chains) and complex RL algorithms (DDPG with GRUs), indicating high math density. It is grounded in empirical testing via simulations and real equity pair trading data, though the backtest-ready implementation details are not fully specified in the summary.

Research Flowchart

  flowchart TD
  A["Research Goal: Optimal Trading<br>with Partial/Latent Info"] --> B["Data & Environment"]
  B --> C{"RL Agent Architecture"}
  
  subgraph B ["Inputs"]
      B1["Ornstein-Uhlenbeck Signal<br>with Regime-Switching"]
      B2["Market Observations"]
  end

  subgraph C ["Core Methodology: DDPG + GRU"]
      direction TB
      C1["GRU: Extract Temporal<br>Latent Features"]
      C2["RL Actor-Critic<br>DDPG Policy Optimization"]
      C1 --> C2
  end

  C --> D{"Algorithm Variants"}
  
  D --> E["hid-DDPG<br>Direct Hidden States"]
  D --> F["prob-DDPG<br>Posterior Regime<br>Probabilities"]
  D --> G["reg-DDPG<br>Signal Forecast"]

  E & F & G --> H["Simulation &<br>Equity Pair Trading"]
  
  H --> I{"Key Findings / Outcomes"}
  I --> J["prob-DDPG: Superior<br>Profit & Interpretability"]
  I --> K["reg-DDPG: Limited Benefit"]
  I --> L["hid-DDPG: Intermediate<br>Performance"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes three Deep Deterministic Policy Gradient (DDPG) algorithms that integrate Gated Recurrent Unit (GRU) networks to solve an optimal trading problem with a mean‑reverting Ornstein‑Uhlenbeck signal and latent regime information.

Source sentence
To address this problem, we propose three Deep Deterministic Policy Gradient (DDPG)–based algorithms that integrate Gated Recurrent Unit (GRU) networks to capture temporal dependencies in the signal.

Assumes 6

  • The trading signal follows an Ornstein‑Uhlenbeck process with regime‑switching dynamics governed by independent Markov chains. model
    Source sentence
    we study an optimal trading problem, where a trading signal follows an Ornstein–Uhlenbeck process with regime‑switching dynamics.
  • Transaction costs are a linear per‑unit fee λ ≥ 0 applied to each trade volume. execution
    Source sentence
    Whenever the trader performs a trade of volume q_t, she pays a transaction cost per unit volume of λ ≥0.
  • The agent can hold inventory within a bounded interval [‑I_max, I_max] and starts with zero inventory. execution
    Source sentence
    the agent can hold inventories within the interval I ∈[‑Imax, Imax] and we always start the testing episodes of trading with S0 = 1 and I0 = 0.
  • The trader is risk‑neutral and maximises the expected discounted sum of rewards. model
    Source sentence
    a risk neutral trader who maximises the expected discounted profits from trading.
Show 2 more
  • The latent parameters (mean reversion, speed, volatility) are filtered from observations of the signal. data
    Source sentence
    The latent parameters driving mean reversion, speed, and volatility are filtered from observations of the signal.
  • The pair‑trading assets are co‑integrated, allowing a stationary linear combination to be used as the trading portfolio. model
    Source sentence
    We see that both tests indicate strong evidence of at least one co‑integrating vector

Data

Recognised sources: LOBSTER

Costs

Modeled Costs are part of the reported headline results.

A per‑unit transaction cost λ ≥ 0 is subtracted from the reward for each trade volume.

Source sentence
Whenever the trader performs a trade of volume q_t, she pays a transaction cost per unit volume of λ ≥0. As a result, the reward r_t from the trading action at time t is r_t(q_t, S_t, S_{t+1}, λ) := I_{t+1} (S_{t+1}‑S_t) ‑λ |q_t| .

Finds 5

  • prob‑DDPG consistently achieves the highest cumulative trading rewards and the most interpretable strategies in both synthetic and real‑world pair‑trading experiments. performance
    Source sentence
    we find that prob‑DDPG achieves superior cumulative rewards and exhibits more interpretable strategies.
  • reg‑DDPG provides limited benefits, offering little improvement over baseline. performance
    Source sentence
    By contrast, reg‑DDPG provides limited benefits.
  • hid‑DDPG yields intermediate performance with less interpretable strategies. performance
    Source sentence
    hid‑DDPG offers intermediate performance with less interpretable strategies.
  • Providing posterior probability estimates of the mean‑reversion regimes leads to the highest trading rewards, confirming the importance of structured information. performance
    Source sentence
    The numerical results show that the agent, when equipped with posterior probability estimates of the mean reversion regimes (which it learns in a first stage), achieves the highest trading rewards.
  • The conclusions about the superiority of prob‑DDPG hold when applied to real market pair‑trading data. performance
    Source sentence
    Finally, we find that these conclusions remain valid when the agent faces trading problems with real market data, when dealing with a pair trading problem for two co‑integrated assets.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 27 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 1 paper in the archive (1 by independent authors)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (37)

  1. On the Optimal Selection of Time-Lag Embedding Dimension for Deep Learning Approaches in Financial Forecasting With Big Data (2026)
  2. Development of a cryptocurrency price prediction model: leveraging GRU and LSTM for Bitcoin, Litecoin and Ethereum (2025)
  3. SA cooperative deep learning model for stock market prediction using deep autoencoder and sentiment 451 analysis (2022)
  4. Reinforcement learning for algorithmic trading (2023)
  5. Deep Q-learning for Nash equilibria: Nash-DQN (2022)
  6. Double deep q-learning for optimal execution (2021)
  7. Trading algorithms with learning in latent alpha models (2019)
  8. Incorporating order-flow into optimal execution (2016)
and 29 more
  1. Dynamic trading with predictable returns and transaction costs (2013)
  2. Continuous control with deep reinforcement learning (2015) — link
  3. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation (2014)
  4. Algorithmic and high-frequency trading (2015)
  5. Deep Reinforcement Learning for Active High Frequency Trading (2023) — link
  6. Dynamic portfolio optimization with inverse covariance clustering (2023) — link
  7. Stock Market Prediction via Deep Learning Techniques: A Survey — → 2212.12717
  8. Using deep learning for price prediction by exploiting stationary limit order book features (2020)
  9. Universal features of price formation in financial markets: perspectives from deep learning (2021)
  10. Deeplob: Deep convolutional neural networks for limit order books (2019)
  11. Deep order flow imbalance: Extracting alpha at multiple horizons from the limit order book (2023)
  12. Deep reinforcement learning for trading (2019) — link
  13. Deep reinforcement learning for automated stock trading: An ensemble strategy (2020)
  14. Recent advances in reinforcement learning in finance (2023)
  15. Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014) — link
  16. Algorithmic trading: winning strategies and their rationale (2013)
  17. A new approach to the economic analysis of nonstationary time series and the business cycle (1989) — link
  18. Planning and acting in partially observable stochastic domains (1998)
  19. Optimal control of Markov processes with incomplete state information I (1965)
  20. Multi-agent reinforcement learning: Foundations and modern approaches (2024)
  21. Some applications of stochastic differential equations to optimal nonlinear filtering (1964)
  22. Q-learning for stochastic control under general information structures and non-Markovian environments (2023) — link
  23. Statistical analysis of cointegration vectors (1988)
  24. Estimation and hypothesis testing of cointegration vectors in Gaussian vector autoregressive models (1991)
  25. The optimal control of partially observable Markov processes over a finite horizon (1973)
  26. Markov chains (1998)
  27. Recurrent Neural Networks and Universal Approximation of Bayesian Filters (2023)
  28. Recurrent Networks, Hidden States and Beliefs in Partially Observable Environments (2022)
  29. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling (2014) — link

1 of 37 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.