{"paper_id":"2505.19058","title":"Distributionally Robust Deep Q-Learning","authors":["Chung I Lu","Julian Sester","Aijia Zhang"],"paper_date":"2025-05-25","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2505.19058v1","page_url":"https://thequant.space/flowcharts/distributionally-robust-deep-q-learning/","scores":{"empirical_rigor":7.5,"math_complexity":9,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes a distributionally robust Q‑learning algorithm for continuous state spaces, using a Sinkhorn‑distance ambiguity set and a deep neural network implementation called Robust DQN (RDQN).","quote":"We propose a novel distributionally robust Q-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty.","summary":"We propose a novel distributionally robust Q-learning algorithm for the non-tabular case accounting for continuous state spaces where the state transition of the underlying Markov decision process is subject to model uncertainty. The uncertainty is taken into account by considering the worst-case…","strategy":"Robust DQN (RDQN)"},"assumptions":{"status":"available","items":[{"assumption":"Assumptions 2.4 and 2.6 hold true","type":"model","stated":"explicit","quote":"Assume that Assumptions 2.4 and 2.6 hold true, then for δ ≥0 and ε > 0, the map"},{"assumption":"State space is compact","type":"other","stated":"explicit","quote":"We provide theorectical gaurantees for the existence of solutions when the state space is compact."}]},"datasets":{"status":"available","items":[{"name":"S&P 500 index data","provider":null,"kind":"market-prices","access":"unknown","period":"3 Jan 1995 to 28 Dec 2023","universe":"S&P 500 constituents","quote":"we generate 1,000 paths independent from the training dataset, each of length equal to the S&P 500 data period from 3 Jan 1995 to 28 Dec 2023."}],"known":[],"sample":{"start":"1995-01","end":"2023-12","market":"S&P 500 index","asset_class":"equities","frequency":"daily"},"data_resolution":"daily"},"transaction_costs":{"status":"available","modeled":"yes","headline_net_of_costs":true,"detail":"0.05% transaction cost used in main experiments; also tests with zero cost and 0.25% during training.","quote":"This agent is trained with ε = 0.0025, δ = 0.0001 and a transaction cost of 0.05%.","source":"front_matter","rule":"R3","candidates":{"front_matter":"yes","evidence_card":"yes","evidence_card_quoted":true},"card_detail":"0.05% transaction cost used in main experiments; also tests with zero cost and 0.25% during training.","card_quote":"This agent is trained with ε = 0.0025, δ = 0.0001 and a transaction cost of 0.05%.","meaning":"costs are part of the reported headline results"},"results":{"status":"available","evaluation_type":"trading-pnl","findings":[{"finding":"RDQN outperforms DQN on all average risk‑adjusted metrics despite lower absolute wealth.","kind":"performance","quote":"we observe see a reversal of fortune for the RDQN agent where it is now outperforming the DQN agent on all average risk-adjusted metrics."},{"finding":"With zero transaction cost both agents accumulate significantly more wealth, and RDQN achieves higher Sharpe and Sortino ratios than DQN and the S&P 500 index.","kind":"performance","quote":"... the Sharpe and Sortino ratios are also significantly higher with the RDQN agents outperforming both the DQN agent and the S&P 500 index."},{"finding":"In the simulator, the DQN agent delivers significantly better wealth than the average simulated path.","kind":"performance","quote":"It is clear from the wealth metric, that the DQN agent delivers significantly better results than the average simulated path"},{"finding":"Overall, RDQN shows better adaptation to distributional shift than DQN across all result sets.","kind":"performance","quote":"Overall, we see better adaptation to the distributional shift with the RDQN agent compared to the DQN agent consistently across all three sets of results"}],"reported_performance":{"strategy":"Robust DQN (RDQN)","sharpe":0.52,"sharpe_benchmark":null,"annual_return_pct":null,"total_return_pct":null,"max_drawdown_pct":34.7},"out_of_sample":{"method":"unclear","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":50,"citations":{"cited_by_in_corpus":1,"by_independent_authors":1,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":1,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":true,"code_url":"https://github.com/luchungi/Sinkhorn_RDQN","note":null},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:37:09Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}