{"paper_id":"2311.02088","title":"Combining Deep Learning on Order Books with Reinforcement Learning for Profitable Trading","authors":["Koti S. Jaddu","Paul A. Bilokon"],"paper_date":"2023-10-24","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2311.02088v1","page_url":"https://thequant.space/flowcharts/combining-deep-learning-on-order-books-with-reinforcement-le/","scores":{"empirical_rigor":6,"math_complexity":7.5,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes combining deep learning on order books with reinforcement learning to forecast returns across multiple horizons using order flow imbalance and train three temporal‑difference models for five financial instruments to generate trading signals.","quote":"The following work focuses on forecasting returns across multiple horizons using order flow imbalance and training three temporal-difference learning models for five financial instruments to provide trading signals.","summary":"High-frequency trading is prevalent, where automated decisions must be made quickly to take advantage of price imbalances and patterns in price action that forecast near-future movements. While many algorithms have been explored and tested, analytical methods fail to harness the whole nature of the…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"Using the first ten order‑flow‑imbalance (OFI) levels is sufficient for the models.","type":"data","stated":"implied","quote":"The first ten OFI levels were used by Kolm et al. [5] and should be sufficient to carry out this work."},{"assumption":"Commissions applied to every trade match the costs at CTrader FXPro.","type":"execution","stated":"implied","quote":"The commissions applied to every trade match the costs at CTrader FXPro, which from lowest to highest per lot size is GBPUSD/EURUSD, XAUUSD, FTSE100, and DE40."},{"assumption":"A train‑validation‑test split of 8:1:1 is appropriate for the ten‑week OFI dataset.","type":"data","stated":"implied","quote":"Only ten weeks of order flow imbalance data were collected for each instrument and split into 8:1:1 for training, validation, and testing."},{"assumption":"Grid‑search will find optimal hyper‑parameters, and literature‑suggested values are likely best.","type":"model","stated":"implied","quote":"Grid search was used to find optimal parameters for each model, and more likely than not, values suggested in the literature were found to be best."}]},"datasets":{"status":"available","items":[{"name":"CTrader FXPro OFI dataset","provider":"CTrader FXPro","kind":"order-book","access":"proprietary","period":"23/5/2023‑3/7/2023 and 25/7/2023‑21/8/20","universe":"GBPUSD, EURUSD, DE40, FTSE100, XAUUSD","quote":"All data was collected on every working day from 23/5/2023 to 3/7/2023 (6 weeks) and from 25/7/2023 to 21/8/2023 (four weeks)."}],"known":[{"id":"lobster","name":"LOBSTER","provider":"LOBSTER","kind":"order-book","access":"commercial","quote":"As price action in CTrader is reflected by the orders of its users, the networks will be trained on the LOB data from the platform instead of sourcing it from LOBSTER to maximise accuracy.","mentions":3,"used_in":3}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"l2-book"},"transaction_costs":{"status":"available","modeled":"yes","headline_net_of_costs":null,"detail":"Commissions applied to every trade match the costs at CTrader FXPro, ranging from lowest (GBPUSD/EURUSD) to highest (DE40).","quote":"The commissions applied to every trade match the costs at CTrader FXPro, which from lowest to highest per lot size is GBPUSD/EURUSD, XAUUSD, FTSE100, and DE40.","source":"evidence_card","rule":"R2","candidates":{"front_matter":"unclear","evidence_card":"yes","evidence_card_quoted":true},"card_detail":"Commissions applied to every trade match the costs at CTrader FXPro, ranging from lowest (GBPUSD/EURUSD) to highest (DE40).","card_quote":"The commissions applied to every trade match the costs at CTrader FXPro, which from lowest to highest per lot size is GBPUSD/EURUSD, XAUUSD, FTSE100, and DE40.","meaning":"costs are part of the reported headline results"},"results":{"status":"available","evaluation_type":"trading-pnl","findings":[{"finding":"Backtesting with retail costs showed profitability using Q‑Learning on GBPUSD and EURUSD, though forward testing did not replicate the results.","kind":"performance","quote":"Overall, backtesting with retail costs produced promising results, with profitability achieved using Q Learning on GBPUSD and EURUSD, but failed to bring similar results during forward testing."},{"finding":"Alpha‑extraction performance aligns with Kolm et al.’s MLP results, achieving comparable out‑of‑sample R² values.","kind":"predictive-accuracy","quote":"The alpha extraction outcome aligns with Kolm et al.’s [5] results using the MLP architecture."},{"finding":"FTSE100 and DE40 agents approached profitability during training, while XAUUSD agents were not profitable despite lower transaction costs.","kind":"performance","quote":"It is surprising that FTSE100 and DE40 are close to being profitable during training across all agents, while XAUUSD was not profitable at all when having less transaction costs."},{"finding":"DQN agents produced negative daily average profits on all instruments in the test set.","kind":"performance","quote":"Daily Average Profit (£) -151,000 ... -11,500 ... -12,000 ... -4,830 ... -6,750"}],"reported_performance":null,"out_of_sample":{"method":"holdout","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":17,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":true,"code_url":"https://github.com/KotiJaddu/Masters-Project","note":null},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T18:20:25Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}