{"paper_id":"2405.13609","title":"Tackling Decision Processes with Non-Cumulative Objectives using Reinforcement Learning","authors":["Maximilian Nägele","Jan Olle","Thomas Fösel","Remmy Zen","Florian Marquardt"],"paper_date":"2024-05-22","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2405.13609v3","page_url":"https://thequant.space/flowcharts/tackling-decision-processes-with-non-cumulative-objectives-u/","scores":{"empirical_rigor":4,"math_complexity":8.5,"quadrant":"Lab Rats"},"proposes":{"status":"available","statement":"The paper proposes a general mapping that transforms non‑cumulative Markov decision processes (NCMDPs) into standard MDPs, enabling the use of existing reinforcement‑learning methods.","quote":"we introduce a general mapping of NCMDPs to standard MDPs.","summary":"Markov decision processes (MDPs) are used to model a wide variety of applications ranging from game playing over robotics to finance. Their optimal policy typically maximizes the expected sum of rewards given at each step of the decision process. However, a large class of problems does not fit…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The analysis focuses on non‑discounted, episodic MDPs.","type":"model","stated":"explicit","quote":"For simplicity, we discuss only non-discounted, episodic MDPs."},{"assumption":"The mapping works for both deterministic and probabilistic environments.","type":"model","stated":"explicit","quote":"In this work, we described a mapping from a decision process with a general non‑cumulative objective (NCMDP) to a standard Markov decision process (MDP) applicable in deterministic and probabilistic settings."},{"assumption":"The portfolio‑optimization environment provides a history of returns and volatility measures for each index.","type":"data","stated":"explicit","quote":"The states contain a history of returns of each index and different volatility measures."},{"assumption":"Agents trade continuously on daily portfolio allocations without explicit transaction‑cost modeling.","type":"execution","stated":"implied","quote":"Actions are the continuous relative portfolio allocations for each day."},{"assumption":"The Sharpe ratio is computed as mean divided by standard deviation of simple returns.","type":"statistical","stated":"explicit","quote":"STD(˜r0, . . . , ˜rT−1) , where ˜rt = (Pt+1 −Pt)/Pt are the simple returns."}]},"datasets":{"status":"available","items":[{"name":"11 S&P500 sector indices","provider":null,"kind":"market-prices","access":"unknown","period":"2006-2021","universe":"11 sector indices","quote":"We perform experiments on an environment as described by Sood et al. (2023) where an agent trades on the 11 different S&P500 sector indices between 2006 and 2021."}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"daily"},"transaction_costs":{"status":"available","modeled":"no","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R1","candidates":{"front_matter":"no","evidence_card":"unclear","evidence_card_quoted":false},"meaning":"a trading or execution result is reported without costs"},"results":{"status":"available","evaluation_type":"trading-pnl","findings":[{"finding":"The SHARPE algorithm that directly maximizes the exact Sharpe ratio significantly outperforms both the differential Sharpe ratio and the cumulative final‑Sharpe baselines.","kind":"performance","quote":"the SHARPE algorithm significantly outperforms the other algorithms."},{"finding":"In the lunar‑lander control task, the non‑cumulative MAXVELPPO agent achieves a better trade‑off between reward and maximum speed than the cumulative FINALPPO agent.","kind":"performance","quote":"the non‑cumulative MAXVELPPO agent can find a better trade‑off than the cumulative FINALPPO agent."},{"finding":"The proposed mapping improves both final performance and training efficiency compared to standard MDP approaches.","kind":"performance","quote":"Our approach improves both final performance and training efficiency compared to relying on standard MDPs."}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":12,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T18:09:02Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}