Paper: arXiv 2401.06179
Authors: Sina Montazeri, Akram Mirzaeinia, Haseebullah Jumakhan, Amir Mirzaeinia
Abstract
The published MLP-based DRL in finance has difficulties in learning the dynamics of the environment when the action scale increases. If the buying and selling increase to one thousand shares, the MLP agent will not be able to effectively adapt to the environment. To address this, we designed a CNN agent that concatenates the data from the last ninety days of the daily feature vector to create the CNN input matrix. Our extensive experiments demonstrate that the MLP-based agent experiences a loss corresponding to the initial environment setup, while our designed CNN remains stable, effectively learns the environment, and leads to an increase in rewards.
Complexity vs Empirical Score
- Math Complexity: 3.0/10
- Empirical Rigor: 8.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper relies on standard deep learning architectures (MLP/CNN) and basic RL formulations (MDP) with minimal complex derivations, but it details a full trading environment, specific datasets, and reports performance metrics, indicating a high degree of empirical testing.
Research Flowchart
flowchart TD
A["Research Goal:<br>Address Action Scaling<br>in Finance DRL"] --> B["Input Data:<br>90-day Market Features"]
B --> C["Methodology:<br>CNN vs MLP Agents"]
C --> D["Computation:<br>Deep Reinforcement Learning<br>with Scaled Actions"]
D --> E{"Comparison"}
E --> F["Outcome: MLP Agent<br>Loss & Instability"]
E --> G["Outcome: CNN Agent<br>Stability & Reward Increase"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a convolutional neural network (CNN) architecture as a deep reinforcement learning (DRL) agent for stock trading that can handle large, continuous action spaces up to 1000 shares.
Source sentence
In this work, we design a CNN architecture as a DRL agent to effectively discern salient features from financial data and achieve high rewards with wide-ranging buy/sell actions.
Assumes 5
- The trading environment can be modeled as a Markov Decision Process with state represented as a matrix of the last 90 days of data. model
Source sentence
We have updated the environment developed by Finrl to enable the environment state representation in the form of a matrix. Our modified environment initializes the state matrix with data from the initial 90-day period of the dataset.
- The turbulence exclusion mechanism is not activated, i.e., the agent learns during turbulent periods. market
Source sentence
In contrast, we chose not to activate the turbulence exclusion and instead closely monitored and took action during this period as well as other period of time.
- Fixed transaction cost percentages from the FinRL framework are applied to all trades. execution
Source sentence
We employ identical fixed cost percentages as those developed in the FinRL framework.
- Reward values are scaled down to one for learning stability. model
Source sentence
This reward is scaled down to one as it returns from the environment to the agent to help the learning process
Show 1 more
- PPO and A2C are the chosen DRL algorithms, assumed to be appropriate for the task. model
Source sentence
We select PPO and A2C which are commonly used and show higher performance than other approaches.
Data
- Dow Jones 30 companies price data market-prices public
Source sentence
We procure the stock prices of thirty companies listed in the Dow Jones from January 2015 through September 2023. This period encompasses the pandemic
Costs
Modeled
Identical fixed cost percentages as those developed in the FinRL framework are applied to all trades.
Source sentence
We employ identical fixed cost percentages as those developed in the FinRL framework.
Finds 4
- The CNN agent remains stable and learns effectively, achieving increased rewards compared to the MLP agent when action size is scaled to 1000 shares. performance
Source sentence
... while our designed CNN remains stable, effectively learns the environment, and leads to an increase in rewards.
- CNN achieves higher accumulated rewards and lower trading cost than MLP across both PPO and A2C algorithms. performance
Source sentence
Figure 4 presents a comparison of key training statistics between the CNN and MLP agents. As shown, the CNN model achieved better accumulated rewards, while maintaining a lower cost compared to the MLP architecture.
- CNN attains superior risk‑adjusted returns (higher Sharpe ratio) than MLP for both PPO and A2C. performance
Source sentence
Figure 5 highlights the superior risk-adjusted returns (Sharpe ratio) attained by our proposed CNN across both PPO and A2C algorithms.
- CNN learns the environment and increases its reward while the action size is increased to one thousand. performance
Source sentence
Fig. 3 shows the learning reward collected by our CNN and prior MLP agents when two PPO and A2C algorithms are used. As it is demonstrated, our CNN agent is able to learn the environment and increase its reward while the action size is increased to one thousand.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 49 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (17)
- Proximal policy optimization algorithms (2017) — link
- Asynchronous methods for deep reinforcement learning (2016)
- Imagenet classification with deep convolutional neural networks (2012)
- FinRL-Meta: Market environments and benchmarks for data-driven financial reinforcement learning (2022)
- Deep reinforcement learning in quantitative algorithmic trading: A review (2021) — link
- Wharton — link
- Performance functions and reinforcement learning for trading systems and portfolios (1998)
- Cryptocurrency portfolio management with deep reinforcement learning (2017)
and 9 more
- Reinforcement learning: An introduction (2018)
- CNNPred: CNN-based stock market prediction using several data sources (2018) — link
- Time series analysis: forecasting and control (2015)
- Forecasting seasonals and trends by exponentially weighted moving averages (2004)
- Evaluating multiple classifiers for stock price direction prediction (2015)
- Machine learning for quantitative finance: fast derivative pricing, hedging and fitting (2018)
- Financial time series forecasting using support vector machines (2003)
- Deep learning with long short-term memory networks for financial market predictions (2018)
- CNN-DRL with Shuffled Features in Finance — → 2402.03338
1 of 17 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.