{"paper_id":"2309.14615","title":"Gray-box Adversarial Attack of Deep Reinforcement Learning-based Trading Agents","authors":["Foozhan Ataiefard","Hadi Hemmati"],"paper_date":"2023-09-26","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2309.14615v1","page_url":"https://thequant.space/flowcharts/gray-box-adversarial-attack-of-deep-reinforcement-learning-b/","scores":{"empirical_rigor":7,"math_complexity":5.5,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes a gray-box framework for generating adversarial samples against Deep RL trading agents, using an adversary with a hybrid deep neural network policy.","quote":"we propose a gray-box framework to create adversarial samples for Deep RL trading agents similar to trading in a real stock market.","summary":"In recent years, deep reinforcement learning (Deep RL) has been successfully implemented as a smart agent in many systems such as complex games, self-driving cars, and chat-bots. One of the interesting use cases of Deep RL is its application as an automated stock trading agent. In general, any…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The adversary has no access to the victim's source code, policy architecture, DNN weights, or training algorithms; only the current market state and the victim's chosen action are observable.","type":"model","stated":"explicit","quote":"The gray-box assumption is that, the trading agents’ source code, policy architecture, DNN weights, and training algorithms are all unknown for the adversary. The only accessible data is the current state of the market and the decision of the trading agent (its chosen trading action in that given state, which is public"},{"assumption":"The reward function R = (Balance + P −ˆP) × α + |π(a|S) −π(ˆa| ˆS)| adequately captures the adversary’s objective.","type":"model","stated":"explicit","quote":"We propose R as reward function for our adversary agent: R = (Balance + P −ˆP) × α + |π(a|S) −π(ˆa| ˆS)|"},{"assumption":"Training the adversary using only 4 days of stock market data (8% of test data) is sufficient to learn effective attacks.","type":"data","stated":"explicit","quote":"We use Categorical Cross Entropy loss to train the adversary using only 4 days of stock market data (8% of test data)."},{"assumption":"OpenAI Gym simulated environment can faithfully represent the interaction between the victim and adversary agents.","type":"other","stated":"implied","quote":"OpenAI Gym provides a perfect framework for training a vast range of agents on different datasets and simulations. However, it does not offer an environment for agents to bet against each other."}]},"datasets":{"status":"available","items":[{"name":"4 days of stock market data","provider":null,"kind":"market-prices","access":"unknown","period":null,"universe":null,"quote":"We use Categorical Cross Entropy loss to train the adversary using only 4 days of stock market data (8% of test data)."},{"name":"Simulated trading market configurations","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"On average, over three simulated trading market configurations, the adversary policy proposed in this research is able to reduce the reward values by 214.17%."}],"known":[{"id":"abides","name":"ABIDES simulator","provider":"ABIDES","kind":"synthetic","access":"public","quote":"As we use OpenAI Gym’s implementations for our experiments, we have integrated ABIDES into the Gym environment.","mentions":5,"used_in":2}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"synthetic"},"transaction_costs":{"status":"not_extracted","modeled":"unclear","headline_net_of_costs":null,"detail":null,"quote":null,"source":"none","rule":"R0","candidates":{"front_matter":"unclear","evidence_card":"unclear","evidence_card_quoted":false},"meaning":"the classifier was not confident"},"results":{"status":"available","evaluation_type":"unclear","findings":[{"finding":"The adversary reduces the reward value by 214.17%, corresponding to a 139.4% reduction in baseline trader profits, 93.7% for an ensemble trader, and 85.5% for an industrial trader.","kind":"performance","quote":"immediate profits (on average over the three market configs per trader) by 139.4% for baseline trader, 93.7% for ensemble, and 85.5% for industry trader in its weakest attack."},{"finding":"The adversary consumes significantly less budget than the victims, with budget consumption ratios of 427.77%, 187.16%, and 66.97% respectively.","kind":"performance","quote":"while consuming significantly less budget than the victims (427.77%, 187.16%, and 66.97%, respectively)."},{"finding":"Kolmogorov‑Smirnov tests reject the null hypothesis that natural and attack reward distributions are identical (p‑values < 0.05).","kind":"statistical","quote":"Since all of the p-values are extremely smaller than 0.05 which means the distance between D(Rnatural) and D(Rattack) are calculated with confidence, we can safely claim that the defined Null hypotheses from equation (5) has been rejected."},{"finding":"The adversary’s portfolio loss per episode is lower than the victim’s cumulative return loss, indicating a cost‑effective attack.","kind":"performance","quote":"To answer this RQ, we first look at the losses of the victims (CR) vs. the adversary’s (Adversary Portfolio Loss), in table IV."}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":21,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T18:21:44Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}