Paper: arXiv 2309.14615
Authors: Foozhan Ataiefard, Hadi Hemmati
Abstract
In recent years, deep reinforcement learning (Deep RL) has been successfully implemented as a smart agent in many systems such as complex games, self-driving cars, and chat-bots. One of the interesting use cases of Deep RL is its application as an automated stock trading agent. In general, any automated trading agent is prone to manipulations by adversaries in the trading environment. Thus studying their robustness is vital for their success in practice. However, typical mechanism to study RL robustness, which is based on white-box gradient-based adversarial sample generation techniques (like FGSM), is obsolete for this use case, since the models are protected behind secure international exchange APIs, such as NASDAQ. In this research, we demonstrate that a “gray-box” approach for attacking a Deep RL-based trading agent is possible by trading in the same stock market, with no extra access to the trading agent. In our proposed approach, an adversary agent uses a hybrid Deep Neural Network as its policy consisting of Convolutional layers and fully-connected layers. On average, over three simulated trading market configurations, the adversary policy proposed in this research is able to reduce the reward values by 214.17%, which results in reducing the potential profits of the baseline by 139.4%, ensemble method by 93.7%, and an automated trading software developed by our industrial partner by 85.5%, while consuming significantly less budget than the victims (427.77%, 187.16%, and 66.97%, respectively).
Complexity vs Empirical Score
- Math Complexity: 5.5/10
- Empirical Rigor: 7.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper applies advanced deep reinforcement learning concepts like actor-critic networks and policy gradients, but lacks heavy mathematical derivations; it demonstrates strong empirical rigor through multi-market simulations, performance metrics against industrial baselines, and a public replication package.
Research Flowchart
flowchart TD A["Research Goal:<br>How to audit Deep RL trading<br>agents without model access?"] --> B["Method: Gray-box Adversarial<br>Attack via Market Participation"] B --> C["Data/Input:<br>Public Market Data (NASDAQ)<br>Simulation Environment"] C --> D["Computational Process:<br>1. Baseline Victim RL Agents<br>2. Hybrid CNN-DRL Adversary<br>3. Co-simulation in Market"] D --> E["Key Findings:<br>Adversary reduced victim rewards<br>by 214% (Avg)<br>Profits reduced up to 139.4%<br>with lower capital usage"] E --> F["Outcome:<br>Proved Gray-box attacks feasible<br>without model access,<br>highlighting critical security risks<br>in automated trading systems"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a gray-box framework for generating adversarial samples against Deep RL trading agents, using an adversary with a hybrid deep neural network policy.
Source sentence
we propose a gray-box framework to create adversarial samples for Deep RL trading agents similar to trading in a real stock market.
Assumes 4
- The adversary has no access to the victim's source code, policy architecture, DNN weights, or training algorithms; only the current market state and the victim's chosen action are observable. model
Source sentence
The gray-box assumption is that, the trading agents’ source code, policy architecture, DNN weights, and training algorithms are all unknown for the adversary. The only accessible data is the current state of the market and the decision of the trading agent (its chosen trading action in that given state, which is public
- The reward function R = (Balance + P −ˆP) × α + |π(a|S) −π(ˆa| ˆS)| adequately captures the adversary’s objective. model
Source sentence
We propose R as reward function for our adversary agent: R = (Balance + P −ˆP) × α + |π(a|S) −π(ˆa| ˆS)|
- Training the adversary using only 4 days of stock market data (8% of test data) is sufficient to learn effective attacks. data
Source sentence
We use Categorical Cross Entropy loss to train the adversary using only 4 days of stock market data (8% of test data).
- OpenAI Gym simulated environment can faithfully represent the interaction between the victim and adversary agents. other
Source sentence
OpenAI Gym provides a perfect framework for training a vast range of agents on different datasets and simulations. However, it does not offer an environment for agents to bet against each other.
Data
- 4 days of stock market data market-prices unknown
Source sentence
We use Categorical Cross Entropy loss to train the adversary using only 4 days of stock market data (8% of test data).
- Simulated trading market configurations synthetic unknown
Source sentence
On average, over three simulated trading market configurations, the adversary policy proposed in this research is able to reduce the reward values by 214.17%.
ABIDES simulator
Costs
Unclear
Finds 4
- The adversary reduces the reward value by 214.17%, corresponding to a 139.4% reduction in baseline trader profits, 93.7% for an ensemble trader, and 85.5% for an industrial trader. performance
Source sentence
immediate profits (on average over the three market configs per trader) by 139.4% for baseline trader, 93.7% for ensemble, and 85.5% for industry trader in its weakest attack.
- The adversary consumes significantly less budget than the victims, with budget consumption ratios of 427.77%, 187.16%, and 66.97% respectively. performance
Source sentence
while consuming significantly less budget than the victims (427.77%, 187.16%, and 66.97%, respectively).
- Kolmogorov‑Smirnov tests reject the null hypothesis that natural and attack reward distributions are identical (p‑values < 0.05). statistical
Source sentence
Since all of the p-values are extremely smaller than 0.05 which means the distance between D(Rnatural) and D(Rattack) are calculated with confidence, we can safely claim that the defined Null hypotheses from equation (5) has been rejected.
- The adversary’s portfolio loss per episode is lower than the victim’s cumulative return loss, indicating a cost‑effective attack. performance
Source sentence
To answer this RQ, we first look at the losses of the victims (CR) vs. the adversary’s (Adversary Portfolio Loss), in table IV.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 21 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.