Paper: arXiv 2411.08637
Authors: Sven Goluža, Tomislav Kovačević, Stjepan Begušić, Zvonko Kostanjčar
Abstract
The sequential nature of decision-making in financial asset trading aligns naturally with the reinforcement learning (RL) framework, making RL a common approach in this domain. However, the low signal-to-noise ratio in financial markets results in noisy estimates of environment components, including the reward function, which hinders effective policy learning by RL agents. Given the critical importance of reward function design in RL problems, this paper introduces a novel and more robust reward function by leveraging imitation learning, where a trend labeling algorithm acts as an expert. We integrate imitation (expert’s) feedback with reinforcement (agent’s) feedback in a model-free RL algorithm, effectively embedding the imitation learning problem within the RL paradigm to handle the stochasticity of reward signals. Empirical results demonstrate that this novel approach improves financial performance metrics compared to traditional benchmarks and RL agents trained solely using reinforcement feedback.
Complexity vs Empirical Score
- Math Complexity: 7.0/10
- Empirical Rigor: 6.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper integrates advanced RL/IL concepts with formal MDP definitions and algorithmic modifications, reflecting high math complexity. Empirical results on intraday futures data and risk-adjusted metrics show backtest-ready implementation and data-heavy validation.
Research Flowchart
flowchart TD A["Research Goal: <br>Robust RL for <br>Noisy Financial Markets"] --> B["Methodology: <br>Imitation-Enhanced Reward <br>(Expert + RL Feedback)"] B --> C["Input: <br>Asset Trading Data <br>(Equities)"] C --> D["Computational Process: <br>Model-Free RL Algorithm <br>using Hybrid Reward Signal"] D --> E["Outcome: <br>Improved Financial Metrics <br>vs. Benchmarks"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper introduces a novel reward function that combines reinforcement feedback with imitation feedback from a trend‑labeling expert.
Source sentence
The key contribution of this paper is the introduction of a novel reward function that leverages IL and uses a trend labeling algorithm as an expert.
Assumes 6
- Finite‑horizon episodic setup where the agent trades over one day and no discounting is applied (γ=1). model
Source sentence
In this research, we assume a finite-horizon setup, where the agent makes decisions over one trading day, so we do not apply discounting of future rewards since the discount factor γ is fixed to 1.
- Positions are liquidated by the end of the day to avoid overnight risk. execution
Source sentence
The MDP was developed for an episodic intraday trading task, where the agent makes minute-level decisions throughout the trading day and liquidates positions by the end of the day to mitigate overnight risk.
- Trades are executed at the open price of the subsequent minute. execution
Source sentence
We assume that trades are executed at the open price of the subsequent minute.
- Unit‑size positions are used and market impact is negligible. execution
Source sentence
Given that our trading strategy involves unit size positions exclusively, any market impact is considered negligible and thus does not affect price movement in our model.
Show 2 more
- Commission costs are a fixed percentage of the traded price. market
Source sentence
The commission, denoted as c, is calculated as a percentage (denoted as ϕ) of the traded price.
- The expert is a deterministic trend‑labeling algorithm that provides oracle labels per minute. model
Source sentence
Four different sets of labels obtained using the oracle
Data
GC, ES, CL futures contracts · futures · intraday
- Futures dataset (GC, CL, ES) market-prices unknown
Source sentence
We use a futures dataset that includes three assets historically known for their liquidity and sufficient order flow: GC (Gold), CL (Crude Oil), and ES (S&P 500). The dataset consists of OHLCV data sampled at one-minute intervals during the most liquid trading hours, from Monday to Friday, 09:30 to 17:00 ET.
Costs
Modeled
Commission c = ϕ * pO_{t+1} * |a_t|, with ϕ chosen from {0.5,1,2,3,4,5,10,20} bps; expert commission ϑ also grid‑searched in bps.
Source sentence
... We specifically optimize the commission fee ϑ in the imitation feedback and the commission fee ϕ in the reinforcement feedback. We perform a grid search, using a search space consisting of the values {0.5, 1, 2, 3, 4, 5, 10, 20} bps for each hyperparameter.
Finds 3
Sharpe 1.28 (benchmark 0.99) · annual return 11.0% · max drawdown 6.6%
- RIF reduces the number of trades for ES dramatically (200 vs 396 for RL) while increasing average holding periods, indicating more stable decisions. performance
Source sentence
The number of trades made by the proposed approach combining reinforcement and imitation feedback (RIF) is drastically reduced for ES, modestly for CL, and slightly increased for GC, when compared to the reinforcement feedback approach (RF).
- RIF achieves higher win‑rates and larger average positive returns than the benchmarks across the three futures contracts. performance
Source sentence
Moreover, the winrate is generally increased when compared to the buy and hold trades and is improved over the RF winrate in 2 out of 3 test assets.
- Overall, the method significantly outperforms traditional RL agents trained solely with reinforcement feedback and the passive buy‑and‑hold strategy. performance
Source sentence
The experimental results showed that the proposed method significantly outperforms traditional RL agents trained solely with reinforcement feedback and the passive buy and hold strategy.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 33 related papers). Coverage is partial, so this is neither replication nor refutation.
1 paper in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (41)
- Optimal Trend Labeling in Financial Time Series (2023) — link
- Reinforcement learning: An introduction (2018)
- Mastering the game of go without human knowledge (2017)
- Human-level control through deep reinforcement learning (2015)
- Deep learning (2016)
- Efficient capital markets: A review of theory and empirical work (1970)
- Effect of labeling algorithms on financial performance metrics (2022)
- Deep robust reinforcement learning for practical algorithmic trading (2019) — link
and 33 more
- Adaptive quantitative trading: An imitative deep reinforcement learning approach (2020)
- Imitation Learning for Financial Applications (2023)
- Proximal policy optimization algorithms (2017)
- Deep reinforcement learning for trading (2020) — link
- Deep reinforcement learning for automated stock trading: An ensemble strategy (2020)
- High-dimensional continuous control using generalized advantage estimation (2015)
- Pseudomathematics and financial charlatanism: The effects of backtest over fitting on out-of-sample performance (2014) — link
- Adam: A method for stochastic optimization (2014)
- Technical analysis of the financial markets: A comprehensive guide to trading methods and applications (1999)
- Does algorithmic trading improve liquidity? (2011) — link
- Deep order flow imbalance: Extracting alpha at multiple horizons from the limit order book (2023) — link
- Algorithms for Inverse Reinforcement Learning (2000)
- Noise (1986) — link
- Deep direct reinforcement learning for financial signal representation and trading (2016) — link
- DeepScalper: A risk-aware reinforcement learning framework to capture fleeting intraday trading opportunities (2022)
- Reinforcement Learning with a Corrupted Reward Channel (2017) — link
- Reward Estimation for Variance Reduction in Deep Reinforcement Learning (2018) — link
- Reinforcement learning with perturbed rewards (2020) — link
- A distributional perspective on reinforcement learning (2017)
- Deep reinforcement learning from human preferences (2017)
- Inverse reward design (2017)
- Imitation Learning: Progress, Taxonomies and Challenges (2024) — link
- SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards (2020)
- Imitation Learning by Reinforcement Learning (2022)
- RLIF: Interactive Imitation Learning as Reinforcement Learning (2024)
- Random expert distillation: Imitation learning via expert policy support estimation (2019)
- Generative adversarial imitation learning (2016)
- Learning robust rewards with adversarial inverse reinforcement learning (2018) — link
- No-regret reductions for imitation learning and structured prediction (2011)
- HG-DAgger: Interactive imitation learning with human experts (2019) — link
- Fleet-DAgger: Interactive Robot Fleet Learning with Scalable Human Supervision (2023)
- Goal-conditioned imitation learning (2019)
- Deep reinforcement learning with positional context for intraday trading — → 2406.08013
1 of 41 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.