Paper: arXiv 2604.00031

Authors: Nabeel Ahmad Saidd

Abstract

Applying reinforcement learning (RL) to foreign exchange (Forex) trading remains challenging because realistic environments, well-defined reward functions, and expressive action spaces must be satisfied simultaneously, yet many prior studies rely on simplified simulators, single scalar rewards, and restricted action representations, limiting both interpretability and practical relevance. This paper presents a modular RL framework designed to address these limitations through three tightly integrated components: a friction-aware execution engine that enforces strict anti-lookahead semantics, with observations at time t, execution at time t+1, and mark-to-market at time t+1, while incorporating realistic costs such as spread, commission, slippage, rollover financing, and margin-triggered liquidation; a decomposable 11-component reward architecture with fixed weights and per-step diagnostic logging to enable systematic ablation and component-level attribution; and a 10-action discrete interface with legal-action masking that encodes explicit trading primitives while enforcing margin-aware feasibility constraints. Empirical evaluation on EURUSD focuses on learning dynamics rather than generalization and reveals strongly non-monotonic reward interactions, where additional penalties do not reliably improve outcomes; the full reward configuration achieves the highest training Sharpe (0.765) and cumulative return (57.09 percent). The expanded action space increases return but also turnover and reduces Sharpe relative to a conservative 3-action baseline, indicating a return-activity trade-off under a fixed training budget, while scaling-enabled variants consistently reduce drawdown, with the combined configuration achieving the strongest endpoint performance.

Complexity vs Empirical Score

  • Math Complexity: 3.5/10
  • Empirical Rigor: 6.0/10 (capped from 6.5: no out-of-sample test; see penalty flags)
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper provides a highly practical framework for RL in Forex trading by focusing on realistic execution dynamics, transaction frictions, and action-space design rather than advanced mathematical proofs. Its empirical focus on learning dynamics and reward attribution offers useful practical insights, though its scope is somewhat limited by relying primarily on training performance on a single currency pair.

Research Flowchart

  flowchart TD
    A["Research Goal: Realism & Interpretability in Forex RL"] --> B["Data Input: EUR/USD Market Data"]
    B --> C["Friction-Aware Engine<br>(Anti-lookahead, Spread, Slippage, Rollover)"]
    C --> D["RL Framework Design<br>(10-Action Space + 11-Component Reward Architecture)"]
    D --> E["Training & Ablation Analysis<br>(Fixed Weights & Legal Action Masking)"]
    E --> F["Key Findings:<br>- Full reward yields highest Sharpe (0.765) & Return (57.09%)<br>- Non-monotonic reward interactions observed<br>- Expanded actions increase returns but trade off Sharpe"]

Cited References (29)

  1. Human-level control through deep reinforcement learning (2015)
  2. Deep reinforcement learning with double Q -learning (2016)
  3. Dueling network architectures for deep reinforcement learning (2016)
  4. Deep direct reinforcement learning for financial signal representation and trading (2017)
  5. A deep reinforcement learning framework for the financial portfolio management problem (2017) — link
  6. An application of deep reinforcement learning to algorithmic trading (2021)
  7. FinRL : A deep reinforcement learning library for automated stock trading in quantitative finance (2021) — link
  8. Reinforcement learning applied to forex trading (2018)
and 21 more
  1. Reinforcement learning in financial markets (2019)
  2. Deep LSTM with reinforcement learning layer for financial trend prediction in FX high frequency trading systems (2019)
  3. Deep reinforcement learning for automated stock trading: An ensemble strategy (2019)
  4. Proximal policy optimization algorithms (2017)
  5. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor (2018)
  6. A distributional perspective on reinforcement learning (2017)
  7. Reinforcement Learning: An Introduction (2018)
  8. Learning to trade via direct reinforcement (2001)
  9. Testing different reinforcement learning configurations for financial trading: Introduction and applications (2012)
  10. Policy invariance under reward transformations: Theory and application to shaped reward reinforcement learning (1999)
  11. A closer look at invalid action masking in policy gradient algorithms (2020) — link
  12. OpenAI Gym (2016) — link
  13. The S harpe Ratio (1994)
  14. Performance measurement in a downside risk framework (1994)
  15. Portfolio selection (1952)
  16. Rainbow: Combining improvements in deep reinforcement learning (2018)
  17. Prioritized experience replay (2016)
  18. Deep learning with long short-term memory networks for financial market predictions (2018)
  19. Empirical properties of asset returns: Stylized facts and statistical issues (2001)
  20. Deep reinforcement learning for trading (2020)
  21. Limit order books (2013)

0 of 29 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.