{"paper_id":"2402.12049","title":"Reinforcement Learning for Optimal Execution when Liquidity is Time-Varying","authors":["Andrea Macrì","Fabrizio Lillo"],"paper_date":"2024-02-19","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2402.12049v2","page_url":"https://thequant.space/flowcharts/reinforcement-learning-for-optimal-execution-when-liquidity/","scores":{"empirical_rigor":4,"math_complexity":7.5,"quadrant":"Lab Rats"},"proposes":{"status":"available","statement":"The paper proposes a Double Deep Q‑Learning (DDQL) algorithm that learns optimal liquidation strategies under time‑varying liquidity.","quote":"we show that the use of Double Deep Q-learning, a form of Reinforcement Learning based on neural networks, is able to learn optimal trading policies when liquidity is time-varying.","summary":"Optimal execution is an important problem faced by any trader. Most solutions are based on the assumption of constant market impact, while liquidity is known to be dynamic. Moreover, models with time-varying liquidity typically assume that it is observable, despite the fact that, in reality, it is…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The trader must unwind an initial inventory q0 within a fixed time window [0,T] divided into N equal sub‑intervals.","type":"execution","stated":"explicit","quote":"We will consider the problem of a trader who wants to unwind a portfolio of q0 stocks within a time window [0,T]."},{"assumption":"The impact of trades on price follows the Almgren‑Chriss linear market impact model.","type":"model","stated":"explicit","quote":"The Almgren and Chriss [3] (A&C) model is used as the baseline model."},{"assumption":"The agent is risk‑neutral (risk‑aversion parameter λ = 0), implying a TWAP strategy in the baseline.","type":"model","stated":"explicit","quote":"In this paper the trader (or agent) is assumed to be risk neutral with λ = 0, thus a risk‑neutral agent in discrete time adopts the TWAP strategy."},{"assumption":"Permanent and temporary impact coefficients can be deterministic or stochastic functions of time.","type":"market","stated":"explicit","quote":"We consider a setting where liquidity, as described by the temporary and permanent impact coefficient, are not constant but time varying."},{"assumption":"Liquidity (impact coefficients) is latent and not directly observable by the algorithm.","type":"other","stated":"explicit","quote":"models with time‑varying liquidity typically assume that it is observable, despite the fact that, in reality, it is latent and hard to measure in real time."}]},"datasets":{"status":"none_found","items":[],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"synthetic"},"transaction_costs":{"status":"available","modeled":"yes","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R1","candidates":{"front_matter":"yes","evidence_card":"unclear","evidence_card_quoted":false},"meaning":"costs are part of the reported headline results"},"results":{"status":"available","evaluation_type":"execution-cost","findings":[{"finding":"The DDQL agent learns the optimal policy when an analytical solution is available, matching the benchmark costs.","kind":"theoretical","quote":"Using extensive numerical experiments, we show that the trained algorithm learns the optimal policy when the analytical solution is available"},{"finding":"In the constant‑impact setting, DDQL achieves ΔP&L of –0.455 bps (features Q,T) and –0.225 bps (features Q,T,S), comparable to the Almgren‑Chriss model.","kind":"performance","quote":"Table 2: Constant impact. Comparison between the costs of the DDQL agent and of the A&C model. ΔP&L values are reported in basis points."},{"finding":"With stochastic low‑λ impact, DDQL with features Q,T,S attains an average IS of 0.2572 and std.dev. 0.007, improving over the baseline.","kind":"performance","quote":"Table 10: Stochastic impact. Comparison between DDQL agent and theoretical solutions. ΔP&L values are reported in basis points."},{"finding":"Under stochastic high‑λ impact, DDQL with features Q,T,S reduces average IS to 0.3158 (std.dev. 0.0112) versus the theoretical 0.5017, showing superior performance.","kind":"performance","quote":"Table 11: Stochastic impact. Comparison between DDQL agent and theoretical solutions. ΔP&L values are reported in basis points."},{"finding":"When only inventory and time are provided as features, the DDQL agent reproduces a TWAP‑like strategy with ΔP&L of –0.4 bps and low variance.","kind":"performance","quote":"The ΔP&L shows a difference of just −0.4 basis points and a standard deviation of only two basis points."}],"reported_performance":null,"out_of_sample":{"method":"unclear","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[],"contradicts":[],"qualifies":[],"compared_with":19,"citations":{"cited_by_in_corpus":8,"by_independent_authors":6,"how_cited":{"used_as_baseline":0,"uses_method_or_data":1,"extends":0,"discusses_findings":0,"mentions":3,"unclassified":4},"independent_papers":[{"paper":{"paper_id":"2610.03369","title":"Mixture-of-Experts for Cryptocurrency Order Execution: Training Stability, Tail Risk, and Failure Modes","paper_date":"2026-10-02","authors":["Alexander Ardaiz","Varun Budati","Ali Habibnia"],"empirical_rigor":8,"page_url":"https://thequant.space/flowcharts/mixture-of-experts-for-cryptocurrency-order-execution-training-stability-tail/"},"how":"uses_method_or_data","quote":null}],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T18:14:31Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}