Paper: arXiv 2508.03910
Authors: Caio de Souza Barbosa Costa, Anna Helena Reali Costa
Abstract
Recently, reinforcement learning has achieved remarkable results in various domains, including robotics, games, natural language processing, and finance. In the financial domain, this approach has been applied to tasks such as portfolio optimization, where an agent continuously adjusts the allocation of assets within a financial portfolio to maximize profit. Numerous studies have introduced new simulation environments, neural network architectures, and training algorithms for this purpose. Among these, a domain-specific policy gradient algorithm has gained significant attention in the research community for being lightweight, fast, and for outperforming other approaches. However, recent studies have shown that this algorithm can yield inconsistent results and underperform, especially when the portfolio does not consist of cryptocurrencies. One possible explanation for this issue is that the commonly used state normalization method may cause the agent to lose critical information about the true value of the assets being traded. This paper explores this hypothesis by evaluating two of the most widely used normalization methods across three different markets (IBOVESPA, NYSE, and cryptocurrencies) and comparing them with the standard practice of normalizing data before training. The results indicate that, in this specific domain, the state normalization can indeed degrade the agent’s performance.
Complexity vs Empirical Score
- Math Complexity: 5.0/10
- Empirical Rigor: 6.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper presents moderate mathematical formulation in its reinforcement learning setup and portfolio equations, while demonstrating strong empirical rigor by testing multiple normalization methods across three distinct financial markets (IBOVESPA, NYSE, cryptocurrencies) and measuring performance impact.
Research Flowchart
flowchart TD
A["Research Goal: Evaluate<br>State Normalization Methods<br>for RL Portfolio Optimization"] --> B["Data Collection:<br>3 Markets (IBOVESPA, NYSE, Crypto)"]
B --> C["Methodology: Compare<br>2 Normalization Methods<br>vs. Standard Pre-normalization"]
C --> D["Computational Process:<br>Train RL Agent<br>using Policy Gradient Algorithm"]
D --> E{"Evaluate Agent<br>Performance across Markets"}
E --> F["Key Finding 1:<br>State Normalization Degrades<br>Agent Performance"]
E --> G["Key Finding 2:<br>Standard Pre-normalization<br>is Superior in this Domain"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper evaluates two common state‑normalization methods versus normalizing input data across three markets and shows that normalizing the data, not the RL state, improves portfolio‑optimization performance.
Source sentence
This article demonstrates empirically that, in the portfolio optimization domain, to normalize the input data instead of the RL agent’s state improves the performance of the agent not only in the stock market but also in the cryptocurrency market.
Assumes 5
- The agent is trained on historical daily price data from 2011‑11‑11 to 2019‑12‑31 and tested on the calendar year 2020. market
Source sentence
In the NYSE and IBOVESPA portfolios, the agent is trained in the period between 2011/11/11 and 2019/12/31 and tested in the year of 2020 (from January 1st to December 31st).
- A flat commission fee of 0.25 % is applied on each rebalancing transaction. execution
Source sentence
Commission rate 0.0025 Rate of comission fee applied when rebalancing the portfolio.
- State representation uses a time window of 50 steps. model
Source sentence
Time window 50 Size (in steps) of the time window considered in the state representation.
- The markets considered are NYSE (stable), IBOVESPA (more volatile), and a cryptocurrency basket (most volatile). market
Source sentence
In comparison to NYSE, this market is considerably more volatile.
Show 1 more
- Daily closing, high and low prices are sufficient to represent the market state. data
Source sentence
In both cases, the data is composed of daily closing, high and low prices of the components of the portfolio.
Data
- NYSE and IBOVESPA daily price data market-prices public
Source sentence
For NYSE and IBOVESPA, the price of the stocks is taken from Yahoo Finance website.
- Cryptocurrency daily price dataset market-prices public
Source sentence
The cryptocurrencies’ historical prices, on the other hand, were obtained from a dataset available in Kaggle.
Costs
Modeled
A flat commission fee of 0.25 % per trade is applied when rebalancing the portfolio.
Source sentence
Commission rate 0.0025 Rate of comission fee applied when rebalancing the portfolio.
Finds 3
- State normalization can degrade the RL agent’s performance, while normalizing the input data consistently yields higher final accumulative portfolio values across all three markets. performance
Source sentence
The results indicate that, in this specific domain, the state normalization can indeed degrade the agent’s performance.
- Data normalization achieved the highest max FAPV in NYSE (1.74), IBOVESPA (2.47) and Crypto (2.23) compared to other normalization variants. performance
Source sentence
TABLE V MAX FAPV FOR EACH MARKET AND NORMALIZATION METHOD.
- More volatile markets benefit more from data normalization, as the agent avoids information loss inherent to state‑normalization methods. performance
Source sentence
The results of the experiments showed that the more volatile a market is, the more it benefits from the usage of the data normalization since the agent does not suffer from the information loss inherent to the state normalization methods.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 13 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (157)
- TensorFlow : A System for Large-Scale Machine Learning (2016)
- The Buy and Hold Investment Strategy : Building and Evaluating Passive Portfolios from Iraq Stock Exchange (2023) — link
- Continuous Control with Stacked Deep Dynamic Recurrent Reinforcement Learning for Portfolio Optimization (2020) — link
- Deep Reinforcement Learning and Mean-Variance Strategies for Responsible Portfolio Optimization (2024) — link
- Optuna: A Next-generation Hyperparameter Optimization Framework (2019) — link
- An Adaptive Portfolio Trading System: A Risk-Return Portfolio Optimization Using Recurrent Reinforcement Learning with Expected Maximum Drawdown (2017) — link
- Selimamrouni/ Deep-Portfolio-Management-Reinforcement-Learning : V2.0 (2022) — link
- Hidden-Layer Configurations in Reinforcement Learning Models for Stock Portfolio Optimization (2025) — link
and 149 more
- Ranaroussi/Quantstats (2023)
- Quality Attributes in Edge Computing for the Internet of Things : A Systematic Mapping Study (2021) — link
- Deep Reinforcement Learning for Portfolio Management of Markets with a Dynamic Number of Assets (2021) — link
- Deep Reinforcement Trading with Predictable Returns (2023) — link
- Fantastic MASs and Where to Find Them: First Results and Lesson Learned (2023)
- OpenAI Gym (2016) — link
- Algorithmic and High-Frequency Trading (2015)
- A Primer on Partially Observable Markov Decision Processes ( POMDPs ) (2021) — link
- Reinforcement Learning with Convolutional Reservoir Computing (2020) — link
- XGBoost : A Scalable Tree Boosting System (2016) — link
- Graph Time-series Modeling in Deep Learning : A Survey (2024) — link
- On the Properties of Neural Machine Translation : Encoder-Decoder Approaches (2014) — link
- Xception: Deep Learning with Depthwise Separable Convolutions (2017) — link
- A Study of Low-Volatility PortfolioConstruction Methods (2014) — link
- Intention Is Choice with Commitment (1990) — link
- Support-Vector Networks (1995) — link
- A Comparison of the Basic Principles and Behavioural Aspects of Akka , JaCaMo and Jade Development Frameworks (2018)
- POE : A General Portfolio Optimization Environment for FinRL (2023) — link
- RLPortfolio : Reinforcement Learning for~ Financial Portfolio Optimization (2025) — link
- Temporal fusion transformers para a precifica \c c ~a o de ativos financeiros (2022)
- Portfolio Constructions in Cryptocurrency Market: A CVaR-based Deep Reinforcement Learning Approach (2023) — link
- K- Nearest Neighbour Classifiers : 2nd Edition (with Python Examples) (2022) — link
- Deep Direct Reinforcement Learning for Financial Signal Representation and Trading (2017) — link
- Reinforcement Learning Applied to Trading Systems : A Survey (2022) — link
- Artificial Intelligence Applied to Stock Market Trading : A Review (2021) — link
- Limitations of Physics Informed Machine Learning for Nonlinear Two-Phase Transport in Porous Media (2020) — link
- Deep Echo State Network ( DeepESN ): A Brief Survey (2020) — link
- Continuous-Time Reinforcement Learning Approach for Portfolio Management with Time Penalization (2019) — link
- Bayesian Data Analysis (2015) — link
- A Survey on Software Agent Architectures (2013)
- Generative Adversarial Nets (2014)
- A Brief Review of Portfolio Optimization Techniques (2022) — link
- Gym-Mtsim (2021)
- Gym-Anytrading (2023)
- The Unreasonable Effectiveness of Data (2009) — link
- Recent Advances in Reinforcement Learning in Finance (2023) — link
- Dynamic Neural Networks : A Survey (2021) — link
- Stock Portfolio Management by Using Fuzzy Ensemble Deep Reinforcement Learning Algorithm (2023) — link
- Array Programming with NumPy (2020) — link
- Using Echo State Networks to Approximate Value Functions for Control (2021) — link
- Neural Networks : A Comprehensive Foundation (1999)
- Literature Review: Machine Learning Techniques Applied to Financial Market Prediction (2019) — link
- Rainbow: Combining Improvements in Deep Reinforcement Learning (2017) — link
- Distilling the Knowledge in a Neural Network (2015) — link
- Long Short-Term Memory (1997) — link
- Multilayer Feedforward Networks Are Universal Approximators (1989) — link
- Application of Evolutionary Computation for Rule Discovery in Stock Algorithmic Trading: A Literature Review (2015) — link
- Deep Reinforcement Learning for Stock Portfolio Optimization by Connecting with Modern Portfolio Theory (2023) — link
- A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem (2017) — link
- Deep Reinforcement Learning for Portfolio Selection (2024) — link
- ZhengyaoJiang / PGPortfolio (2024)
- A Mean-VaR Based Deep Reinforcement Learning Framework for Practical Algorithmic Trading (2023) — link
- An Asynchronous Advantage Actor-Critic Reinforcement Learning Method for Stock Selection and Portfolio Management (2018) — link
- Gradient Theory of Optimal Flight Paths (1960) — link
- Text Mining for Market Prediction: A Systematic Review (2014) — link
- Semi- Supervised Classification with Graph Convolutional Networks (2017) — link
- Global Market Portfolio 2023 (2023)
- The Stability , Scalability and Performance of Multi-agent Systems (1998) — link
- ChatGPT : A Meta-Analysis after 2.5 Months (2023) — link
- Investment Diversification as a Strategy for Reducing Investment Risk (2018) — link
- Deep Reinforcement Learning (2018) — link
- Understanding and Addressing Quality Attributes of Microservices Architecture: A Systematic Literature Review (2021) — link
- Online Portfolio Management via Deep Reinforcement Learning with High-Frequency Data (2023) — link
- A Deep Reinforcement Learning Framework For Financial Portfolio Management — → 2409.08426
- Adversarial Deep Reinforcement Learning in Portfolio Management (2018) — link
- Contrastive Learning and Reward Smoothing for Deep Portfolio Management (2023) — link
- Continuous Control with Deep Reinforcement Learning (2015) — link
- Dynamic Portfolio Rebalancing through Reinforcement Learning (2022) — link
- Multiagent-Based Deep Reinforcement Learning for Risk-Shifting Portfolio Management (2022) — link
- Practical Deep Reinforcement Learning Approach for Stock Trading (2018) — link
- ElegantRL-Podracer : Scalable and Elastic Library for Cloud-Native Deep Reinforcement Learning (2022) — link
- FinRL : Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance (2022) — link
- FinRL-Meta : Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning (2022) — link
- Multi-Agent Deep Reinforcement Learning Algorithm with Trend Consistency Regularization for Portfolio Management (2022) — link
- MACD Indicator Explained , with Formula , Examples , and Limitations
- MACD Indicator for Trading - Tutorial and Examples
- On the Maximum Drawdown of a Brownian Motion (2004) — link
- Systematic Review on Reinforcement Learning in the Field of Fintech — → 2305.07466
- Portfolio Selection (1952) — link
- Developing a Risk Management Approach Based on Reinforcement Training in the Formation of an Investment Portfolio (2023) — link
- Engineering Multi-Agent Systems : State of Affairs and the Road Ahead (2020) — link
- Using Deep Reinforcement Learning with Hierarchical Risk Parity for Portfolio Optimization (2023) — link
- Asynchronous Methods for Deep Reinforcement Learning (2016) — link
- Data Augmentation: A Comprehensive Survey of Modern Approaches (2022) — link
- Advancing Investment Frontiers : Industry-grade Deep Reinforcement Learning for Portfolio Optimization (2024) — link
- Does Reinforcement Learning Outperform Deep Learning and Traditional Portfolio Optimization Models in Frontier and Developed Financial Markets? (2023) — link
- MetaTrader : An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization (2022) — link
- A Novel Multi-Source Information-Fusion Predictive Framework Based on Deep Neural Networks for Accuracy Enhancement in Stock Market Prediction (2021) — link
- Attention U-Net : Learning Where to Look for the Pancreas (2018) — link
- ChatGPT (2023)
- An Intelligent Financial Portfolio Trading Strategy Using Deep Q-learning (2020) — link
- Twin-System Recurrent Reinforcement Learning for Optimizing Portfolio Strategy (2024) — link
- PyTorch : An Imperative Style , High-Performance Deep Learning Library (2019) — link
- Are `` Market Neutral ’’ Hedge Funds Really Market Neutral ? (2009) — link
- Scikit-Learn: Machine Learning in Python (2018) — link
- Acceleration of Stochastic Approximation by Averaging (1992) — link
- Stable- Baselines3 : Reliable Reinforcement Learning Implementations (2021)
- A Modelling of Novel Approach for Stock Portfolio Management Using Unsupervised Algorithm and Deep Reinforcement Technique (2023) — link
- Relative Strength Index ( RSI ) Indicator Explained With Formula
- Normalized Online Learning (2013) — link
- Normalized Online Learning (2013) — link
- Introduction to Probability Models (2014) — link
- Curriculum Learning Empowered Reinforcement Learning for Graph-Based Portfolio Management: Performance Optimization and Comprehensive Analysis (2024) — link
- FINBERT-PT-BR: An 'a lise de sentimentos de textos em portugu ^e s referentes ao mercado financeiro (2022)
- Modeling Relational Data with Graph Convolutional Networks (2017) — link
- Trust Region Policy Optimization (2015) — link
- Proximal Policy Optimization Algorithms (2017) — link
- The Sharpe Ratio (1994) — link
- A Multi-Scale Temporal Feature Aggregation Convolutional Neural Network for Portfolio Management (2019) — link
- GPM : A Graph Convolutional Network Based Reinforcement Learning Framework for Portfolio Management (2022) — link
- Mastering the Game of Go without Human Knowledge (2017) — link
- Investigating the Impact of Data Normalization on Classification Performance (2020) — link
- Portfolio Optimization Using Reinforcement Learning : A Study of Implementation of Learning to Optimize (2023) — link
- Smart Toys Market Size & Share Analysis - Industry Research Report - Growth Trends
- Financial Portfolio Optimization with Online Deep Reinforcement Learning and Restricted Stacked Autoencoder— DeepBreath (2020) — link
- Deep Graph Convolutional Reinforcement Learning for Financial Portfolio Management – DeepPocket (2021) — link
- From Deterministic to Stochastic: An Interpretable Stochastic Model-Free Reinforcement Learning Framework for Portfolio Optimization (2023) — link
- Reinforcement Learning-Based Portfolio Optimization with Deterministic State Transition (2025) — link
- Combining Transformer Based Deep Reinforcement Learning with Black-Litterman Model for Portfolio Optimization — → 2402.16609
- GraphSAGE with Deep Reinforcement Learning for Financial Portfolio Optimization (2024) — link
- Reinforcement Learning : An Introduction (2018)
- Going Deeper with Convolutions (2015) — link
- Pandas-Dev/Pandas: Pandas (2023) — link
- Gymnasium (2023) — link
- Optimizing Automated Trading Systems with Deep Reinforcement Learning (2023) — link
- Some Mathematical Notes on Three-Mode Factor Analysis (1966) — link
- UCI Machine Learning Repository : Adult Data Set
- Deep Reinforcement Learning Methods for Portfolio Management : A Survey
- Moving Average Convergence-Divergence ( MACD ) Trading Rule : An Application in Nepalese Stock Market " NEPSE " (2020) — link
- Deep Reinforcement Learning with Double Q-learning (2015) — link
- Attention Is All You Need (2023) — link
- One-Hot-Encoding, o Que 'e ? (2018)
- Risk-Sensitive Policies for Portfolio Management (2022) — link
- Portfolio Trading System of Digital Currencies: A Deep Reinforcement Learning with Multidimensional Attention Gating Mechanism (2020) — link
- Tianshou: A Highly Modularized Deep Reinforcement Learning Library (2022) — link
- What Is EMA ?- Exponential Moving Average - Fidelity
- Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (1992) — link
- Risk- Aware Reinforcement Learning for~ Multi-Period Portfolio Selection (2023) — link
- World Internet Users Statistics and 2023 World Population Stats
- A Comprehensive Survey on Graph Neural Networks (2021) — link
- Portfolio Management System in Equity Market Neutral Using Reinforcement Learning (2021) — link
- Relation- Aware Transformer for Portfolio Policy Learning (2020) — link
- Reinforcement Learning for POMDP Environments Using State Representation with Reservoir Computing (2022) — link
- Qlib: An AI-oriented Quantitative Investment Platform (2020) — link
- Deep Reinforcement Learning for Automated Stock Trading (2023)
- Deep Reinforcement Learning for Portfolio Management (2023) — link
- Model-Based Deep Reinforcement Learning for Dynamic Portfolio Optimization (2019) — link
- Financial Time Series Forecasting Using CNN and Transformer — → 2304.04912
- Imitation Learning : Progress , Taxonomies and Challenges (2022) — link
4 of 157 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.