{"paper_id":"2411.08637","title":"Robot See, Robot Do: Imitation Reward for Noisy Financial Environments","authors":["Sven Goluža","Tomislav Kovačević","Stjepan Begušić","Zvonko Kostanjčar"],"paper_date":"2024-11-13","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2411.08637v1","page_url":"https://thequant.space/flowcharts/robot-see-robot-do-imitation-reward-for-noisy-financial-en/","scores":{"empirical_rigor":6,"math_complexity":7,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper introduces a novel reward function that combines reinforcement feedback with imitation feedback from a trend‑labeling expert.","quote":"The key contribution of this paper is the introduction of a novel reward function that leverages IL and uses a trend labeling algorithm as an expert.","summary":"The sequential nature of decision-making in financial asset trading aligns naturally with the reinforcement learning (RL) framework, making RL a common approach in this domain. However, the low signal-to-noise ratio in financial markets results in noisy estimates of environment components…","strategy":"RIF (reinforcement + imitation feedback)"},"assumptions":{"status":"available","items":[{"assumption":"Finite‑horizon episodic setup where the agent trades over one day and no discounting is applied (γ=1).","type":"model","stated":"explicit","quote":"In this research, we assume a finite-horizon setup, where the agent makes decisions over one trading day, so we do not apply discounting of future rewards since the discount factor γ is fixed to 1."},{"assumption":"Positions are liquidated by the end of the day to avoid overnight risk.","type":"execution","stated":"explicit","quote":"The MDP was developed for an episodic intraday trading task, where the agent makes minute-level decisions throughout the trading day and liquidates positions by the end of the day to mitigate overnight risk."},{"assumption":"Trades are executed at the open price of the subsequent minute.","type":"execution","stated":"explicit","quote":"We assume that trades are executed at the open price of the subsequent minute."},{"assumption":"Unit‑size positions are used and market impact is negligible.","type":"execution","stated":"explicit","quote":"Given that our trading strategy involves unit size positions exclusively, any market impact is considered negligible and thus does not affect price movement in our model."},{"assumption":"Commission costs are a fixed percentage of the traded price.","type":"market","stated":"explicit","quote":"The commission, denoted as c, is calculated as a percentage (denoted as ϕ) of the traded price."},{"assumption":"The expert is a deterministic trend‑labeling algorithm that provides oracle labels per minute.","type":"model","stated":"implied","quote":"Four different sets of labels obtained using the oracle"}]},"datasets":{"status":"available","items":[{"name":"Futures dataset (GC, CL, ES)","provider":null,"kind":"market-prices","access":"unknown","period":"October 2022 to March 2024 (test), earli","universe":"Gold, Crude Oil, S&P 500 futures","quote":"We use a futures dataset that includes three assets historically known for their liquidity and sufficient order flow: GC (Gold), CL (Crude Oil), and ES (S&P 500). The dataset consists of OHLCV data sampled at one-minute intervals during the most liquid trading hours, from Monday to Friday, 09:30 to 17:00 ET."}],"known":[],"sample":{"start":null,"end":null,"market":"GC, ES, CL futures contracts","asset_class":"futures","frequency":"intraday"},"data_resolution":"intraday-bars"},"transaction_costs":{"status":"available","modeled":"yes","headline_net_of_costs":true,"detail":"Commission c = ϕ * pO_{t+1} * |a_t|, with ϕ chosen from {0.5,1,2,3,4,5,10,20} bps; expert commission ϑ also grid‑searched in bps.","quote":"... We specifically optimize the commission fee ϑ in the imitation feedback and the commission fee ϕ in the reinforcement feedback. We perform a grid search, using a search space consisting of the values {0.5, 1, 2, 3, 4, 5, 10, 20} bps for each hyperparameter.","source":"front_matter","rule":"R3","candidates":{"front_matter":"yes","evidence_card":"yes","evidence_card_quoted":true},"card_detail":"Commission c = ϕ * pO_{t+1} * |a_t|, with ϕ chosen from {0.5,1,2,3,4,5,10,20} bps; expert commission ϑ also grid‑searched in bps.","card_quote":"... We specifically optimize the commission fee ϑ in the imitation feedback and the commission fee ϕ in the reinforcement feedback. We perform a grid search, using a search space consisting of the values {0.5, 1, 2, 3, 4, 5, 10, 20} bps for each hyperparameter.","meaning":"costs are part of the reported headline results"},"results":{"status":"available","evaluation_type":"trading-pnl","findings":[{"finding":"RIF reduces the number of trades for ES dramatically (200 vs 396 for RL) while increasing average holding periods, indicating more stable decisions.","kind":"performance","quote":"The number of trades made by the proposed approach combining reinforcement and imitation feedback (RIF) is drastically reduced for ES, modestly for CL, and slightly increased for GC, when compared to the reinforcement feedback approach (RF)."},{"finding":"RIF achieves higher win‑rates and larger average positive returns than the benchmarks across the three futures contracts.","kind":"performance","quote":"Moreover, the winrate is generally increased when compared to the buy and hold trades and is improved over the RF winrate in 2 out of 3 test assets."},{"finding":"Overall, the method significantly outperforms traditional RL agents trained solely with reinforcement feedback and the passive buy‑and‑hold strategy.","kind":"performance","quote":"The experimental results showed that the proposed method significantly outperforms traditional RL agents trained solely with reinforcement feedback and the passive buy and hold strategy."}],"reported_performance":{"strategy":"RIF (reinforcement + imitation feedback)","sharpe":1.276,"sharpe_benchmark":0.992,"annual_return_pct":10.982,"total_return_pct":null,"max_drawdown_pct":6.554},"out_of_sample":{"method":"walk-forward","headline_out_of_sample":true}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":33,"citations":{"cited_by_in_corpus":1,"by_independent_authors":1,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":1},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:54:16Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}