Paper: arXiv 2410.23294
Authors: Vito Alessandro Monaco, Antonio Riva, Luca Sabbioni, Lorenzo Bisi, Edoardo Vittori, Marco Pinciroli, Michele Trapletti, Marcello Restelli
Abstract
In recent years, the popularity of artificial intelligence has surged due to its widespread application in various fields. The financial sector has harnessed its advantages for multiple purposes, including the development of automated trading systems designed to interact autonomously with markets to pursue different aims. In this work, we focus on the possibility of recognizing and leveraging intraday price patterns in the Foreign Exchange market, known for its extensive liquidity and flexibility. Our approach involves the implementation of a Reinforcement Learning algorithm called Fitted Natural Actor-Critic. This algorithm allows the training of an agent capable of effectively trading by means of continuous actions, which enable the possibility of executing orders with variable trading sizes. This feature is instrumental to realistically model transaction costs, as they typically depend on the order size. Furthermore, it facilitates the integration of risk-averse approaches to induce the agent to adopt more conservative behavior. The proposed approaches have been empirically validated on EUR-USD historical data.
Complexity vs Empirical Score
- Math Complexity: 7.5/10
- Empirical Rigor: 6.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper employs a sophisticated Reinforcement Learning algorithm (Fitted Natural Actor-Critic) involving continuous action spaces, risk aversion modeling, and natural gradient theory, indicating high mathematical complexity. It is validated on real historical EUR-USD data with realistic transaction cost modeling, though it lacks specific code or detailed performance metrics in the excerpt, suggesting moderate empirical rigor.
Research Flowchart
flowchart TD
A["Research Goal<br>Identify & leverage intraday FX patterns<br>to optimize automated trading"] --> B["Methodology: Fitted Natural Actor-Critic<br>Reinforcement Learning with continuous actions"]
B --> C["Data & Inputs<br>EUR-USD historical data"]
C --> D["Key Features Implemented"]
D --> E["Computational Process<br>Agent training & execution"]
subgraph D ["Key Features Implemented"]
F["Size-dependent fees<br>Realistic transaction costs"]
G["Risk-aversion modeling<br>Conservative trading behavior"]
end
E --> H["Key Findings & Outcomes"]
H --> I["Effective trading strategy<br>trained on intraday patterns"]
H --> J["Realistic cost modeling<br>via variable order sizes"]
H --> K["Risk-averse agent behavior<br>validated empirically"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes using the Fitted Natural Actor-Critic (FNAC) algorithm with continuous actions to model size‑dependent transaction fees and integrate risk‑averse approaches for Forex trading.
Source sentence
Our approach involves the implementation of a Reinforcement Learning algorithm called Fitted Natural Actor-Critic. This algorithm allows the training of an agent capable of effectively trading by means of continuous actions, which enable the possibility of executing orders with variable trading sizes
Assumes 6
- Transaction costs increase with order size because larger trades face lower immediate liquidity. execution
Source sentence
Indeed, the larger the trade size (in the order of million dollars), the more difficult it becomes to find enough immediate liquidity for the requested volume, resulting in higher transaction costs.
- The fee function depends on order size rather than on time. model
Source sentence
It is important to recall that the fixed/variable nature of the transaction fee we refer to regards the dependence on the order size, not the dependence on time.
- Historical EUR/USD price data from 2018‑2022 is representative for training and testing. data
Source sentence
We first split the available data, which includes market observations from 2018 to 2022, into four parts.
- Risk‑averse behavior can be induced via mean‑volatility and RCVaR risk measures. model
Source sentence
through the use of risk measures (mean‑volatility and RCVaR), the trained agents successfully developed risk‑averse behaviors
Show 2 more
- Continuous action spaces better reflect real trading environments than discrete actions. other
Source sentence
trading is one of those tasks where considering continuous action spaces enables a more realistic representation of the environment and expands the range of viable solutions
- Lower trading frequency improves signal‑to‑noise ratio and generalization. model
Source sentence
As expected, a lower trading frequency ensured a better signal‑to‑noise ratio, which increased the generalization capabilities of the model.
Data
EUR-USD · fx · intraday
- EUR/USD historical price data market-prices unknown
Source sentence
The proposed approaches have been empirically validated on EUR-USD historical data.
Costs
Modeled
Fees are modeled as either fixed or variable, with the variable fee depending on order size.
Source sentence
the fixed/variable nature of the transaction fee we refer to regards the dependence on the order size, not the dependence on time.
Finds 5
- FNAC achieves significantly better performance than all baselines (FQI, PPO, Buy&Hold, Sell&Hold) on out‑of‑sample EUR/USD data. performance
Source sentence
FNAC has proven to achieve significantly better performance compared to all the baselines, included FQI and PPO, for the considered test dataset.
- In continuous‑action mode FNAC performs slightly worse than in discrete‑action mode, showing higher drawdowns and larger return volatility. performance
Source sentence
the proposed approach obtained slightly worse performances in its continuous form w.r.t. the discrete one. For instance, in the last 4 months of 2022, we noticed higher drawdowns and a significant rise in the standard deviation of the cumulative return.
- Risk‑averse FNAC agents reduce reward variability compared to risk‑neutral agents. performance
Source sentence
the trained agents successfully developed risk‑averse behaviors, decreasing the variability of the obtained rewards.
- Variable transaction fees (size‑dependent) can be learned, allowing the agent to limit order volume to balance revenue and costs. performance
Source sentence
Through the integration of a size‑dependent fee function, we were able to train an agent capable of limiting the order volume in order to better exploit the trade‑off between revenues and transaction costs.
- Discrete‑action FNAC with persistence 10 (trading every 10 minutes) outperforms other persistence settings in 2022. performance
Source sentence
the model trained with persistence equal to 10 significantly outperforms the others in 2022.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 50 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (36)
- Policy gradient methods for reinforcement learning with function approximation (1999)
- Reinforcement learning: An introduction (2018)
- Soft actor-critic algorithms and applications (2018) — link
- Continuous deep q-learning with model-based acceleration (2016)
- Learning to trade via direct reinforcement (2001)
- Fitted natural actor-critic: A new algorithm for continuous state-action MDPs (2008)
- Natural actor-critic (2008)
- Information-geometric optimization algorithms: A unifying picture via invariance principles (2017)
and 28 more
- Playing atari with deep reinforcement learning (2013) — link
- Deep reinforcement learning for active high frequency trading (2021) — link
- Markov decision processes: discrete stochastic dynamic programming (2014)
- Risk-averse trust region optimization for reward-volatility reduction (2021)
- Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learning (2020) — link
- Robust forex trading with deep q network (dqn) (2019)
- Trust region policy optimization (2015)
- A deep reinforcement learning approach for automated cryptocurrency trading (2019)
- Model learning for robot control: a survey (2011)
- Optimizing market making using multi-agent reinforcement learning (2018) — link
- Automated trading systems statistical and machine learning methods and hardware implementation: a survey (2019)
- An adaptive portfolio trading system: A risk-return portfolio optimization using recurrent reinforcement learning with expected maximum drawdown (2017)
- Foreign exchange trading: A risk-averse batch reinforcement learning approach (2020)
- Tree-based batch mode reinforcement learning (2005)
- Learning FX trading strategies with FQI and persistent actions (2021)
- HistData.com
- Xgboost: A scalable tree boosting system (2016)
- The truncated normal distribution (2014)
- A comprehensive survey on safe reinforcement learning (2015)
- Risk-Averse Optimization of Reward-based Coherent Risk Measures (2023)
- Markov decision processes with average-value-at-risk criteria (2011)
- Addressing Non-Stationarity in FX Trading with Online Model Selection of Offline RL Experts (2022)
- Reinforcement learning applied to Forex trading (2018)
- Financial trading as a game: A deep reinforcement learning approach (2018) — link
- Deep reinforcement learning for automated stock trading: An ensemble strategy (2020)
- Proximal policy optimization algorithms (2017) — link
- Continuous control with deep reinforcement learning (2015) — link
- Skulls, financial turbulence, and risk management (2010)
0 of 36 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.