Gray-box Adversarial Attack of Deep Reinforcement Learning-based Trading Agents

ArXiv ID: 2309.14615 “View on arXiv”

Authors: Unknown

Abstract

In recent years, deep reinforcement learning (Deep RL) has been successfully implemented as a smart agent in many systems such as complex games, self-driving cars, and chat-bots. One of the interesting use cases of Deep RL is its application as an automated stock trading agent. In general, any automated trading agent is prone to manipulations by adversaries in the trading environment. Thus studying their robustness is vital for their success in practice. However, typical mechanism to study RL robustness, which is based on white-box gradient-based adversarial sample generation techniques (like FGSM), is obsolete for this use case, since the models are protected behind secure international exchange APIs, such as NASDAQ. In this research, we demonstrate that a “gray-box” approach for attacking a Deep RL-based trading agent is possible by trading in the same stock market, with no extra access to the trading agent. In our proposed approach, an adversary agent uses a hybrid Deep Neural Network as its policy consisting of Convolutional layers and fully-connected layers. On average, over three simulated trading market configurations, the adversary policy proposed in this research is able to reduce the reward values by 214.17%, which results in reducing the potential profits of the baseline by 139.4%, ensemble method by 93.7%, and an automated trading software developed by our industrial partner by 85.5%, while consuming significantly less budget than the victims (427.77%, 187.16%, and 66.97%, respectively).

Keywords: Deep Reinforcement Learning (Deep RL), Adversarial Attacks, Convolutional Neural Networks (CNN), Automated Trading, Market Manipulation, Equities

Complexity vs Empirical Score

• Math Complexity: 5.5/10
• Empirical Rigor: 7.0/10
• Quadrant: Holy Grail
• Why: The paper applies advanced deep reinforcement learning concepts like actor-critic networks and policy gradients, but lacks heavy mathematical derivations; it demonstrates strong empirical rigor through multi-market simulations, performance metrics against industrial baselines, and a public replication package.

  flowchart TD
    A["Research Goal:<br>How to audit Deep RL trading<br>agents without model access?"] --> B["Method: Gray-box Adversarial<br>Attack via Market Participation"]
    
    B --> C["Data/Input:<br>Public Market Data (NASDAQ)<br>Simulation Environment"]
    
    C --> D["Computational Process:<br>1. Baseline Victim RL Agents<br>2. Hybrid CNN-DRL Adversary<br>3. Co-simulation in Market"]
    
    D --> E["Key Findings:<br>Adversary reduced victim rewards<br>by 214% (Avg)<br>Profits reduced up to 139.4%<br>with lower capital usage"]
    
    E --> F["Outcome:<br>Proved Gray-box attacks feasible<br>without model access,<br>highlighting critical security risks<br>in automated trading systems"]