{"paper_id":"2410.14504","title":"Reinforcement Learning in Non-Markov Market-Making","authors":["Luca Lalor","Anatoliy Swishchuk"],"paper_date":"2024-10-18","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2410.14504v2","page_url":"https://thequant.space/flowcharts/reinforcement-learning-in-non-markov-market-making/","scores":{"empirical_rigor":4,"math_complexity":8.5,"quadrant":"Lab Rats"},"proposes":{"status":"available","statement":"The paper proposes a deep reinforcement learning framework for optimal market making that uses the Soft Actor‑Critic (SAC) algorithm and focuses on price processes with semi‑Markov and Hawkes jump‑diffusion dynamics.","quote":"We develop a deep reinforcement learning (RL) framework for an optimal market-making (MM) trading problem, specifically focusing on price processes with semi-Markov and Hawkes Jump-Diffusion dynamics.","summary":"We develop a deep reinforcement learning (RL) framework for an optimal market-making (MM) trading problem, specifically focusing on price processes with semi-Markov and Hawkes Jump-Diffusion dynamics. We begin by discussing the basics of RL and the deep RL framework used, where we deployed the…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The midprice is simulated at each time step and a constant one‑tick spread is assumed.","type":"model","stated":"explicit","quote":"We simulated the midprice at each time step and then assumed a constant 1 tick spread.","support":"full"},{"assumption":"Price and order arrivals are assumed to be independent.","type":"model","stated":"explicit","quote":"This might not be too far off reality in some financial markets and some trading regimes (volatility dependent), but often this is not the case and assuming a constant spread could in fact be highly unrealistic in certain markets. • In these types of frameworks, price and order arrivals are assumed to be independent.","support":"full"},{"assumption":"The training and testing environments are identical, ensuring unbiased machine‑learning results.","type":"data","stated":"explicit","quote":"The environment we used under our training and testing sets are the same, which is based on our analysis in section 3, and this ensures unbiased results in machine learning terms.","support":"full"},{"assumption":"Adverse fills are included in the simulation to avoid phantom gains.","type":"execution","stated":"explicit","quote":"In order to tackle this problem, we made sure to include adverse fills in the simulation process as in Lalor and Swishchuk (2024b).","support":"full"},{"assumption":"Orders are assumed to be automatically at the front of the queue in the limit order book.","type":"market","stated":"explicit","quote":"The models in this analysis and the majority of the HFT literature, however, often just assume that orders are automatically at the front of the queue in the LOB.","support":"full"},{"assumption":"The agent’s action space is continuous, avoiding the curse of dimensionality associated with discrete actions.","type":"model","stated":"implied","quote":"However, the most noticeable limita-tion is the agents discrete action space closely related to Bellmans “curse of dimensionality”.","support":"full"}]},"datasets":{"status":"available","items":[{"name":"Synthetic market‑making environment","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"In the training phase, the agent interacts with the trading environment by taking actions based on the policy.","support":"full"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"synthetic"},"transaction_costs":{"status":"not_extracted","modeled":"unclear","headline_net_of_costs":null,"detail":null,"quote":null,"source":"none","rule":"R0","candidates":{"front_matter":"unclear","evidence_card":"unclear","evidence_card_quoted":false},"meaning":"the classifier was not confident"},"results":{"status":"available","evaluation_type":"none","findings":[{"finding":"The agent starts with a negative cumulative reward at the beginning of training.","kind":"negative","quote":"Unsurprisingly, this agent has a negative cumulative reward at the beginning of the training process as it has not been able to figure out the optimal actions to take yet.","support":"full"},{"finding":"Cumulative reward improves steadily as training progresses.","kind":"performance","quote":"As we get through more episodes, this slowly improves.","support":"full"},{"finding":"When adverse fills are excluded, most terminal rewards are positive, though some extreme negative outliers remain.","kind":"performance","quote":"It is quite apparent that the results are mostly positive when adverse fills are excluded, with some heavy left tail results with extreme negative rewards.","support":"full"},{"finding":"When adverse fills are included, terminal rewards remain largely negative.","kind":"negative","quote":"When adverse fills are included, these results are still quite negative.","support":"full"},{"finding":"Testing results show significantly improved cumulative rewards compared to training, with fewer negative tail events.","kind":"performance","quote":"It is quiet obvious that the results have significantly improved in comparison to their respective training counterparts (in terms of the exclusion/inclusion of adverse fills), in particular regarding the left tail events present in the training phase, and the histogram shows that the cumulative rewards obtained are much more positive or less negative overall in each episode.","support":"full"}],"reported_performance":null,"out_of_sample":{"method":"holdout","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":14,"citations":{"cited_by_in_corpus":4,"by_independent_authors":2,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":4},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":2,"extracted_at":"2026-11-14T19:54:30Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}