{"paper_id":"2408.02322","title":"Consistent time travel for realistic interactions with historical data: reinforcement learning for market making","authors":["Vincent Ragel","Damien Challet"],"paper_date":"2024-08-05","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2408.02322v2","page_url":"https://thequant.space/flowcharts/consistent-time-travel-for-realistic-interactions-with-histo/","scores":{"empirical_rigor":5.5,"math_complexity":6.5,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes a generic method called consistent data time travel for offline reinforcement learning in market making, aligning the agent's actions with historical data by adjusting the time index.","quote":"We introduce consistent data time travel for offline RL as a remedy for these problems: instead of using historical data in a sequential way, we argue that one needs to perform time travel in historical data, i.e., to adjust the time index so that both the past state and the influence of the RL agent's action on the sy","summary":"Reinforcement learning works best when the impact of the agent’s actions on its environment can be perfectly simulated or fully appraised from available data. Some systems are however both hard to simulate and very sensitive to small perturbations. An additional difficulty arises when a RL agent is…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The system state Xt can only be modified by a single event αt from a set of possible events.","type":"model","stated":"explicit","quote":"For the sake of simplicity, assume that Xt can only be modified by"},{"assumption":"Only anonymous, agent‑unresolved data are available; the state of each individual agent cannot be inferred.","type":"data","stated":"explicit","quote":"using only anonymous data, which makes it impossible to infer the state of each agent, thus to use data directly"},{"assumption":"Limit order books are highly sensitive to each single event and retain memory of changes for a long time.","type":"market","stated":"explicit","quote":"limit order books are known to be very sensitive to every single change and do remember them for a long time"},{"assumption":"The RL agent’s action can be represented as an effective event αRL_t that may differ from the historical event αt.","type":"model","stated":"explicit","quote":"... decides to take an action aRL_t which results in an effective event αRL_t"},{"assumption":"When the consistency condition holds, the signs of effective events are sufficient for matching historical data.","type":"statistical","stated":"explicit","quote":"Finding effective events with exactly the same sizes δ is unlikely, thus we only consider the signs of the effective events."}]},"datasets":{"status":"none_found","items":[],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"l2-book"},"transaction_costs":{"status":"not_extracted","modeled":"unclear","headline_net_of_costs":null,"detail":null,"quote":null,"source":"none","rule":"R0","candidates":{"front_matter":"unclear","evidence_card":"unclear","evidence_card_quoted":false},"meaning":"the classifier was not confident"},"results":{"status":"available","evaluation_type":"trading-pnl","findings":[{"finding":"The agent’s gain is significantly higher when using data time travel compared to using naive sequential historical data.","kind":"performance","quote":"it turns out that the gain of the agent is significantly higher with data time travel than with naive sequential data"}],"reported_performance":null,"out_of_sample":{"method":"unclear","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":4,"citations":{"cited_by_in_corpus":1,"by_independent_authors":1,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":1},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T18:04:34Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}