Paper: arXiv 2402.16609
Authors: Ruoyu Sun, Angelos Stefanidis, Zhengyong Jiang, Jionglong Su
Abstract
As a model-free algorithm, deep reinforcement learning (DRL) agent learns and makes decisions by interacting with the environment in an unsupervised way. In recent years, DRL algorithms have been widely applied by scholars for portfolio optimization in consecutive trading periods, since the DRL agent can dynamically adapt to market changes and does not rely on the specification of the joint dynamics across the assets. However, typical DRL agents for portfolio optimization cannot learn a policy that is aware of the dynamic correlation between portfolio asset returns. Since the dynamic correlations among portfolio assets are crucial in optimizing the portfolio, the lack of such knowledge makes it difficult for the DRL agent to maximize the return per unit of risk, especially when the target market permits short selling (i.e., the US stock market). In this research, we propose a hybrid portfolio optimization model combining the DRL agent and the Black-Litterman (BL) model to enable the DRL agent to learn the dynamic correlation between the portfolio asset returns and implement an efficacious long/short strategy based on the correlation. Essentially, the DRL agent is trained to learn the policy to apply the BL model to determine the target portfolio weights. To test our DRL agent, we construct the portfolio based on all the Dow Jones Industrial Average constitute stocks. Empirical results of the experiments conducted on real-world United States stock market data demonstrate that our DRL agent significantly outperforms various comparison portfolio choice strategies and alternative DRL frameworks by at least 42% in terms of accumulated return. In terms of the return per unit of risk, our DRL agent significantly outperforms various comparative portfolio choice strategies and alternative strategies based on other machine learning frameworks.
Complexity vs Empirical Score
- Math Complexity: 7.5/10
- Empirical Rigor: 8.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper combines advanced theoretical frameworks (Transformer DRL, Black-Litterman Bayesian model, deterministic policy gradient) with empirical backtesting on real-world US equity data, demonstrating significant performance improvements over benchmarks.
Research Flowchart
flowchart TD A["Research Goal: Enhance DRL for Portfolio Optimization by Integrating<br>Dynamic Correlation Knowledge for Long/Short Strategy"] --> B["Data Input: US Stock Market Data<br>Dow Jones Industrial Average Constituents"] B --> C["Methodology: Hybrid DRL-Black-Litterman Model"] C --> D["Computational Process:<br>Transformer-based DRL Agent Learns Policy<br>to Determine Portfolio Weights using BL Model"] D --> E["Outcome: Dynamically Optimized Long/Short Portfolio<br>Maximizing Return per Unit of Risk"] E --> F["Key Findings:<br>42%+ Higher Accumulated Return<br>Superior Risk-Adjusted Performance vs. Benchmarks"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a hybrid portfolio optimization model that combines a deep reinforcement learning (DRL) agent with the Black‑Litterman (BL) model to learn dynamic asset correlations and implement a long/short strategy.
Source sentence
In this research, we propose a hybrid portfolio optimization model combining the DRL agent and the Black-Litterman (BL) model to enable the DRL agent to learn the dynamic correlation between the portfolio asset returns and implement an efficacious long/short strategy.
Assumes 4
- Zero Market Impact: trading actions do not affect asset price fluctuations. execution
Source sentence
Zero Market Impact: The trading actions made by the agent do not affect the asset price fluctuation.
- Zero Slippage: selected assets have abundant liquidity and orders are executed immediately. execution
Source sentence
Zero Slippage: The selected assets in the portfolio have notably abundant liquidity in the market. Hence, whether the long position trade orders or short selling orders are issued, they can be immediately executed after they are issued.
- Risk‑free rate is assumed to be zero. model
Source sentence
The risk‑free rate is assumed to be zero in our research.
- One share cannot be divided; target quantities are integer via floor function. other
Source sentence
In Equation (4), we apply the floor function so that the target quantity … contains integers since we assume that one share of stock cannot be divided.
Data
US stock market (Dow Jones Industrial Average constituents) · equities
- Dow Jones Industrial Average constituent stocks market-prices public
Source sentence
We use DJIA constituent stocks to construct our portfolio. After removing stocks with missing data, 29 constituent stocks remain.
Costs
Modeled
Commission fee rate, lending rate of cash, lending rate of stock, trading cost and short‑position cost are included in the cash value update equation.
Source sentence
represent the commission fee rate, lending rate of cash, and lending rate of stock.
Finds 4
- The DRL agent outperforms various comparison portfolio strategies and alternative DRL frameworks by at least 42% in accumulated return on US stock data. performance
Source sentence
demonstrate that our DRL agent significantly outperforms various comparison portfolio choice strategies and alternative DRL frameworks by at least 42% in terms of accumulated return.
- The proposed BDA outperforms its two variants by at least 247% in accumulated return and by at least 24% in Sharpe ratio and Sortino‑Tail ratio. performance
Source sentence
The empirical results show that our BDA outperform the variants by at least 247% in terms of AR. In terms of SR and STR, our BDA outperforms the variants by at least 24% in all experiments.
- Using the BL model for weight determination yields better performance than a softmax‑based variant, improving accumulated return by at least 246% and return‑per‑unit‑of‑risk by at least 7%. performance
Source sentence
BDA exhibits better performance in accumulated return and return per unit of risk. Specifically, our BDA outperforms the variant of BDA (BDA-V4) by at least 246% in terms of AR. In terms of SR and STR, our BDA outperforms the variant of BDA (BDA-V4) by at least 7%.
- Extending the trading period to five days improves out‑sample accumulated return and risk‑adjusted performance, with the BDA outperforming a one‑day variant by at least 12% in SR and STR. performance
Source sentence
the empirical results indicate that extending the trading period to five days is necessary.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 29 related papers). Coverage is partial, so this is neither replication nor refutation.
3 papers in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (90)
- Practical machine learning: Forecasting daily financial markets directions (2023) — link
- StockFormer: Learning Hybrid Trading Machines with Predictive Coding (2023) — link
- A time-varying stock portfolio selection model based on optimized PSO-BiLSTM and multi-objective mathematical programming under budget constraints (2023) — link
- Online Hybrid Neural Network for Stock Price Prediction: A Case Study of High-Frequency Stock Trading in the Chinese Market (2023) — link
- Combining deep reinforcement learning with technical analysis and trend monitoring on cryptocurrency markets (2023) — link
- From deterministic to stochastic: an interpretable stochastic model-free reinforcement learning framework for portfolio optimization (2022) — link
- Forecasting turning points in stock price by applying a novel hybrid CNN-LSTM-ResNet model fed by 2D segmented images (2022) — link
- Q-Learning for Continuous-Time Linear Systems: A Data-Driven Implementation of the Kleinman Algorithm (2022) — link
and 82 more
- On the forecasting of multivariate financial time series using hybridization of DCC-GARCH model and multivariate ANNs (2022) — link
- A multi-agent deep reinforcement learning framework for algorithmic trading in financial markets (2022) — link
- FactorVAE: A Probabilistic Dynamic Factor Model Based on Variational Autoencoder for Predicting Cross-Sectional Stock Returns (2022) — link
- Cardinality-constrained portfolio selection via two-timescale duplex neurodynamic optimization (2022) — link
- BERT’s sentiment score for portfolio optimization: a fine-tuned view in Black and Litterman model (2022) — link
- Are Transformers Effective for Time Series Forecasting? (2022) — link
- Adaptive reinforcement learning in control design for cooperating manipulator systems (2022) — link
- Disturbance observer-based adaptive reinforcement learning for perturbed uncertain surface vessels. (2022) — link
- Breaking the Curse of Dimensionality in Multiagent State Space: A Unified Agent Permutation Framework (2022) — link
- Online Actor-critic Reinforcement Learning Control for Uncertain Surface Vessel Systems with External Disturbances (2022) — link
- Recent advances in reinforcement learning in finance (2021) — link
- FinRL: deep reinforcement learning framework to automate trading in quantitative finance (2021) — link
- Cardinality-constrained portfolio selection based on collaborative neurodynamic optimization (2021) — link
- Decentralized Robust Portfolio Optimization Based on Cooperative-Competitive Multiagent Systems (2021) — link
- Adaptive fuzzy modeling of interval-valued stream data and application in cryptocurrencies prediction (2021) — link
- Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting (2021) — link
- DeepTrader: A Deep Reinforcement Learning Approach for Risk-Return Balanced Portfolio Management with Market Conditions Embedding (2021) — link
- Application of Features and Neural Network to Enhance the Performance of Deep Reinforcement Learning in Portfolio Management (2021) — link
- A Deep Residual Shrinkage Neural Network-based Deep Reinforcement Learning Strategy in Financial Portfolio Management (2021) — link
- The Use of Features to Enhance the Capability of Deep Reinforcement Learning for Investment Portfolio Management (2021) — link
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020) — link
- A deep Q-learning portfolio management framework for the cryptocurrency market (2020) — link
- Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States (2020) — link
- Minimax and Biobjective Portfolio Selection Based on Collaborative Neurodynamic Optimization (2020) — link
- A Multi-Scale Temporal Feature Aggregation Convolutional Neural Network for Portfolio Management (2019) — link
- A Deep Reinforcement Learning Approach for Automated Cryptocurrency Trading (2019) — link
- Adaptive Input Representations for Neural Language Modeling (2018) — link
- Addressing Function Approximation Error in Actor-Critic Methods (2018) — link
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018) — link
- Performance evaluation of series and parallel strategies for financial time series forecasting (2017) — link
- Performance evaluation of series and parallel strategies for financial time series forecasting (2017) — link
- Distributional Reinforcement Learning with Quantile Regression (2017) — link
- Proximal Policy Optimization Algorithms (2017) — link
- A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem (2017) — link
- Attention is All you Need (2017) — link
- Layer Normalization (2016) — link
- Asynchronous Methods for Deep Reinforcement Learning (2016) — link
- Dueling Network Architectures for Deep Reinforcement Learning (2015) — link
- Continuous control with deep reinforcement learning (2015) — link
- Can a corporate network and news sentiment improve portfolio optimization using the Black–Litterman model? (2015) — link
- A Black–Litterman asset allocation model under Elliptical distributions (2015) — link
- Learning a Deep Convolutional Network for Image Super-Resolution (2014) — link
- Playing Atari with Deep Reinforcement Learning (2013) — link
- Weighted Moving Average Passive Aggressive Algorithm for Online Portfolio Selection (2013) — link
- On-Line Portfolio Selection with Moving Average Reversion (2012) — link
- CORN: Correlation-driven nonparametric learning approach for portfolio selection (2011) — link
- Quantitative Equity Investing: Techniques and Strategies (2010)
- Optimal Portfolio Choice with Parameter Uncertainty (2007) — link
- Algorithms for portfolio management based on the Newton method (2006) — link
- NONPARAMETRIC KERNEL‐BASED SEQUENTIAL INVESTMENT STRATEGIES (2006) — link
- The Markowitz Optimization Enigma: Is ‘Optimized’ Optimal? (2005) — link
- Can We Learn to Beat the Best Stock (2003) — link
- Value-at-Risk for long and short trading positions (2003) — link
- Modern Investment Management: An Equilibrium Approach (2003)
- The Intuition Behind Black-Litterman Model Portfolios (2002) — link
- The Sveriges Riksbank (Bank of Sweden) Prize in Economic Sciences in Memory of Alfred Nobel 1969–2000 (2001) — link
- On‐Line Portfolio Selection Using Multiplicative Updates (1998) — link
- Reinforcement Learning: A Survey (1996) — link
- Universal portfolios with side information (1996) — link
- The Influence of the Sigmoid Function Parameters on the Speed of Backpropagation Learning (1995) — link
- Global Portfolio Optimization (1992) — link
- Bayes-Stein Estimation for Portfolio Analysis (1986) — link
- Portfolio Selection (1971) — link
- GPM: A graph convolutional network based reinforcement learning framework for portfolio management (2022) — link
- A Framework of Hierarchical Deep Q-Network for Portfolio Management (2021) — link
- Advances in financial machine learning (2018)
- Online portfolio selection: A survey (2012) — link
- Sortino: a ‘sharper’ratio (2013)
- Efficient backdrop (2012)
- Efficient asset management: a practical guide to stock portfolio optimization and asset allocation (2008)
- A step-by-step guide to the Black-Litterman model: Incorporating user-specified confidence levels (2007)
- The adaptive markets hypothesis: Market effi-ciency from an evolutionary perspective (2004)
- Publisher’s Note (2003) — link
- Markowitz Revisited: Mean-Variance Models in Financial Portfolio Analysis (2001) — link
- Theory and methodology of tactical asset allocation (Vol. 65) (2000)
- Universal Portfolios (1996) — link
- Short Selling and Common Stock Prices (1994) — link
- Asset allocation: combining investor views with market equilibrium (1990)
- Bayesian decision problemas and Markov chains (No. 519.233 M3) (1967)
- Machine Learning manuscript No. (will be inserted by the editor) PAMR: Passive Aggressive Mean Reversion Strategy for Portfolio Selection
- (2022) Adaptive optimal tracking control of an underactuated surface vessel using actor–critic reinforcement learning
- del Este, Uruguay, April 10–14, 2000
0 of 90 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.