{"paper_id":"2411.02211","title":"Reinforcement Learning Methods for the Stochastic Optimal Control of an Industrial Power-to-Heat System","authors":["Eric Pilling","Martin Bähr","Ralf Wunderlich"],"paper_date":"2024-11-04","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2411.02211v3","page_url":"https://thequant.space/flowcharts/reinforcement-learning-methods-for-the-stochastic-optimal-co/","scores":{"empirical_rigor":6.5,"math_complexity":8,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes a stochastic optimal control model for a power‑to‑heat system and solves it with classical dynamic programming and reinforcement learning (Q‑learning) to obtain cost‑optimal decision rules.","quote":"For this system, we investigate the cost-optimal operation aiming to minimize the electricity cost from the grid by a suitable system control depending on the available wind power and the amount of stored thermal energy. Since that approach suffers from the curse of dimensionality we also apply reinforcement learning techniques, namely Q-learning, that are able to provide a good approximate solution to the","summary":"The optimal control of sustainable energy supply systems, including renewable energies and energy storage, takes a central role in the decarbonization of industrial systems. However, the use of fluctuating renewable energies leads to fluctuations in energy generation and requires a suitable control…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The heat flow control A is kept constant between two consecutive time grid points (piecewise constant control).","type":"model","stated":"explicit","quote":"(2.1) As already mentioned in Subsection 2.2, we make the following Assumption 2.1 (Piecewise constant control) The control A is kept constant between two consecutive grid points of the time discretization, i.e. A(t) = A(tn) =: An, t ∈[tn,tn+1), n = 0,...,N −1.","support":"full"},{"assumption":"Thermal losses to the environment are neglected.","type":"model","stated":"explicit","quote":"Note, that we neglect thermal losses to the environment.","support":"full"},{"assumption":"Simultaneous charging and discharging of the thermal energy storage is not allowed.","type":"model","stated":"explicit","quote":"(2.9) 2.4.3 Thermal Energy Storage Operational Modes The TES operation can be divided into three operational modes: charging, discharging and idle, where we assume that simultaneous charging and discharging is not allowed.","support":"full"},{"assumption":"Selling excess energy to the grid is optional and controlled by a binary parameter ζ; if allowed, the selling price is reduced by a spread η.","type":"market","stated":"explicit","quote":"Here, ζ ∈{0,1} is a user defined model parameter that indicates if selling is allowed or not. For ζ = 1, energy is fed into the grid for a reduced market price Ssell given by Ssell(t) = S(t)−η(t), where η : [0,tE] →R+ is called spread.","support":"full"},{"assumption":"The mass flow ˙m and waste heat temperature T_LT,in are assumed constant in the base model.","type":"model","stated":"explicit","quote":"By dropping the assumptions of constant mass flow ˙m and waste heat temperature T LT,in, we can extend our model and make it more general.","support":"full"},{"assumption":"The stochastic processes for wind speed and electricity price are modeled with correlation and calibrated to real‑world data.","type":"data","stated":"explicit","quote":"Apart from providing a modeling approach for the stochastic processes that takes into account correlation between wind speed and electricity price, we also calibrated the associated parameters with real-world data (see Supplementary Material G).","support":"full"}]},"datasets":{"status":"available","items":[{"name":"real‑world wind speed and electricity price data","provider":null,"kind":"macro","access":"unknown","period":null,"universe":null,"quote":"Apart from providing a modeling approach for the stochastic processes that takes into account correlation between wind speed and electricity price, we also calibrated the associated parameters with real-world data (see Supplementary Material G).","support":"full"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"intraday-bars"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R4a","candidates":{"front_matter":"n/a","evidence_card":"yes","evidence_card_quoted":true},"card_detail":"Buying cost uses grid price S(t); selling revenue uses reduced price Ssell(t)=S(t)−η(t), where η reflects transaction fees, taxes or discounts.","card_quote":"For ζ = 1, energy is fed into the grid for a reduced market price Ssell given by Ssell(t) = S(t)−η(t), where η : [0,tE] →R+ is called spread. This spread reflects transaction fees, taxes or the willingness of the grid operator to buy energy only at a certain discount on the market price S(t).","meaning":"the paper reports no trading or execution result; the cost the extractor found is a model parameter"},"results":{"status":"available","evaluation_type":"none","findings":[{"finding":"Both dynamic programming and Q‑learning achieve similar approximations of the value function and yield reliable cost‑optimal decision rules.","kind":"performance","quote":"A comparison of the numerical results shows that both methods can achieve similar ap-proximations of the value function and yields reliable cost-optimal decision rules.","support":"full"},{"finding":"Q‑learning provides a faster and more computationally efficient solution compared to dynamic programming, especially for high‑dimensional problems.","kind":"performance","quote":"Although the results of Q-learning differ in some aspects, it offers a faster and computationally more efficient solution for complex control problems. This is especially useful for problems with high-dimensional state and control spaces, where the dynamic programming approach will fail due to the curse of dimensionality.","support":"full"},{"finding":"When extending the model to include additional state and control variables, dynamic programming becomes intractable due to the curse of dimensionality, while Q‑learning remains feasible.","kind":"theoretical","quote":"By dropping the assumptions of constant mass flow ˙m and waste heat temperature T LT,in, we can extend our model and make it more general. Even though the classical backward recur-sion of dynamic programming might become intractable for this extended model due to the curse of dimensionality, we are confident that Q-learning still offers an efficient solution.","support":"full"}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":15,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":2,"extracted_at":"2026-11-14T19:54:12Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}