{"paper_id":"2507.20263","title":"Learning from Expert Factors: Trajectory-level Reward Shaping for Formulaic Alpha Mining","authors":["Junjie Zhao","Chengxi Zhang","Chenkai Wang","Peng Yang"],"paper_date":"2025-07-27","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2507.20263v1","page_url":"https://thequant.space/flowcharts/learning-from-expert-factors-trajectory-level-reward-shapin/","scores":{"empirical_rigor":7.5,"math_complexity":8,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes Trajectory-level Reward Shaping (TLRS), a novel reward shaping method that provides dense intermediate rewards via exact subsequence matching with expert formulas and includes a reward centering mechanism.","quote":"To address this, Trajectory-level Reward Shaping (TLRS), a novel reward shaping method, is proposed.","summary":"Reinforcement learning (RL) has successfully automated the complex process of mining formulaic alpha factors, for creating interpretable and profitable investment strategies. However, existing methods are hampered by the sparse rewards given the underlying Markov Decision Process. This inefficiency…","strategy":"Trajectory-level Reward Shaping (TLRS)"},"assumptions":{"status":"available","items":[{"assumption":"The discount factor γ must be set to 1 for the factor‑mining MDP.","type":"model","stated":"explicit","quote":"TLRS, a simple yet effective approach, is specifically designed for factor‑mining MDPs where the discount factor γ must be 1."},{"assumption":"The expert factor dataset is generated by a model sharing the same network architecture as the current policy model.","type":"model","stated":"explicit","quote":"Assuming the expert factor dataset is generated by a model sharing the same network architecture as the current policy model being optimized, with weights denoted by θe, while the current policy model has weights θ."}]},"datasets":{"status":"available","items":[{"name":"Real asset price dataset Y","provider":null,"kind":"market-prices","access":"unknown","period":null,"universe":null,"quote":"Input: Real asset price dataset Y, Real asset feature dataset X = {X′l}, Initial policy weight θ,"},{"name":"Six major equity indices (CSI300, CSI500, CSI1000, NDX, DJI, SPX)","provider":null,"kind":"market-prices","access":"unknown","period":"2016–2020","universe":"stocks on the Shanghai, Shenzhen, Nasdaq, and NYSE exchanges","quote":"Our dataset spans four years (2016–2020) and covers four exchanges, reflecting a wide spectrum of market conditions and trading stress."}],"known":[],"sample":{"start":null,"end":null,"market":"Chinese stock indices (CSI300, CSI500)","asset_class":"equities","frequency":"daily"},"data_resolution":"daily"},"transaction_costs":{"status":"not_extracted","modeled":"unclear","headline_net_of_costs":null,"detail":null,"quote":null,"source":"none","rule":"R0","candidates":{"front_matter":"unclear","evidence_card":"unclear","evidence_card_quoted":false},"meaning":"the classifier was not confident"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"TLRS improves the Rank Information Coefficient by 9.29% over existing potential‑based shaping algorithms.","kind":"performance","quote":"It improves Rank IC by 9.29% over existing potential-based shaping algorithms."},{"finding":"TLRS reduces time‑complexity with respect to feature dimension from linear to constant (O(N·L) vs O(N·L·d) or O(N·L·d·P)).","kind":"theoretical","quote":"By contrast, TLRS employs exact RPN subsequence matching to eliminate distance‑metric pitfalls, all at O(N·L) time complexity."},{"finding":"TLRS achieves faster and more stable convergence compared to baseline reward‑shaping methods.","kind":"performance","quote":"TLRS achieves faster and more stable convergence, outperforming all other algorithms."},{"finding":"Ablation shows that both trajectory‑level reward shaping and reward centering are essential; removing either degrades performance.","kind":"performance","quote":"Similarly, the lower performance of TLRS no RC demonstrate that reward centering is vital for stabilizing the training process."}],"reported_performance":{"strategy":"Trajectory-level Reward Shaping (TLRS)","sharpe":null,"sharpe_benchmark":null,"annual_return_pct":null,"total_return_pct":null,"max_drawdown_pct":null},"out_of_sample":{"method":"holdout","headline_out_of_sample":true}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":43,"citations":{"cited_by_in_corpus":2,"by_independent_authors":2,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":2,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:31:05Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}