Paper: arXiv 2411.02211

Authors: Eric Pilling, Martin Bähr, Ralf Wunderlich

Abstract

The optimal control of sustainable energy supply systems, including renewable energies and energy storage, takes a central role in the decarbonization of industrial systems. However, the use of fluctuating renewable energies leads to fluctuations in energy generation and requires a suitable control strategy for the complex systems in order to ensure energy supply. In this paper, we consider an electrified power-to-heat system which is designed to supply heat in form of superheated steam for industrial processes. The system consists of a high-temperature heat pump for heat supply, a wind turbine for power generation, a sensible thermal energy storage for storing excess heat and a steam generator for providing steam. If the system’s energy demand cannot be covered by electricity from the wind turbine, additional electricity must be purchased from the power grid. For this system, we investigate the cost-optimal operation aiming to minimize the electricity cost from the grid by a suitable system control depending on the available wind power and the amount of stored thermal energy. This is a decision making problem under uncertainties about the future prices for electricity from the grid and the future generation of wind power. The resulting stochastic optimal control problem is treated as finite-horizon Markov decision process for a multi-dimensional controlled state process. We first consider the classical backward recursion technique for solving the associated dynamic programming equation for the value function and compute the optimal decision rule. Since that approach suffers from the curse of dimensionality we also apply reinforcement learning techniques, namely Q-learning, that are able to provide a good approximate solution to the optimization problem within reasonable time.

Complexity vs Empirical Score

  • Math Complexity: 8.0/10
  • Empirical Rigor: 6.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper uses advanced stochastic optimal control and Markov decision processes with significant mathematical derivation, but also includes calibration of real-world data and numerical experiments with Q-learning, making it both mathematically dense and empirically grounded.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Cost-optimal control of<br>Power-to-Heat System"] --> B["Methodology: Dynamic Programming & Reinforcement Learning"]
  
  B --> C{"Solve Stochastic<br>Optimal Control Problem"}
  C --> D["Approach 1: Classic Dynamic Programming<br>Backward Recursion"]
  C --> E["Approach 2: Reinforcement Learning<br>Q-learning"]
  
  D --> F["Curse of Dimensionality:<br>Computationally Expensive"]
  E --> G["Approximate Solution:<br>Efficient & Scalable"]
  
  F & G --> H["Outcome: Validated Q-learning<br>provides cost-optimal policy"]
  
  H --> I["Key Finding: RL enables<br>real-time control for<br>complex industrial systems"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a stochastic optimal control model for a power‑to‑heat system and solves it with classical dynamic programming and reinforcement learning (Q‑learning) to obtain cost‑optimal decision rules.

Source sentence
For this system, we investigate the cost-optimal operation aiming to minimize the electricity cost from the grid by a suitable system control depending on the available wind power and the amount of stored thermal energy. Since that approach suffers from the curse of dimensionality we also apply reinforcement learning techniques, namely Q-learning, that are able to provide a good approximate solution to the

Assumes 6

  • The heat flow control A is kept constant between two consecutive time grid points (piecewise constant control). model
    Source sentence
    (2.1) As already mentioned in Subsection 2.2, we make the following Assumption 2.1 (Piecewise constant control) The control A is kept constant between two consecutive grid points of the time discretization, i.e. A(t) = A(tn) =: An, t ∈[tn,tn+1), n = 0,...,N −1.
  • Thermal losses to the environment are neglected. model
    Source sentence
    Note, that we neglect thermal losses to the environment.
  • Simultaneous charging and discharging of the thermal energy storage is not allowed. model
    Source sentence
    (2.9) 2.4.3 Thermal Energy Storage Operational Modes The TES operation can be divided into three operational modes: charging, discharging and idle, where we assume that simultaneous charging and discharging is not allowed.
  • Selling excess energy to the grid is optional and controlled by a binary parameter ζ; if allowed, the selling price is reduced by a spread η. market
    Source sentence
    Here, ζ ∈{0,1} is a user defined model parameter that indicates if selling is allowed or not. For ζ = 1, energy is fed into the grid for a reduced market price Ssell given by Ssell(t) = S(t)−η(t), where η : [0,tE] →R+ is called spread.
Show 2 more
  • The mass flow ˙m and waste heat temperature T_LT,in are assumed constant in the base model. model
    Source sentence
    By dropping the assumptions of constant mass flow ˙m and waste heat temperature T LT,in, we can extend our model and make it more general.
  • The stochastic processes for wind speed and electricity price are modeled with correlation and calibrated to real‑world data. data
    Source sentence
    Apart from providing a modeling approach for the stochastic processes that takes into account correlation between wind speed and electricity price, we also calibrated the associated parameters with real-world data (see Supplementary Material G).

Data

  • real‑world wind speed and electricity price data macro unknown
    Source sentence
    Apart from providing a modeling approach for the stochastic processes that takes into account correlation between wind speed and electricity price, we also calibrated the associated parameters with real-world data (see Supplementary Material G).

Costs

Not applicable The paper reports no trading or execution result; the cost the extractor found is a model parameter.

The paper-text extraction found a cost parameter inside a model (Buying cost uses grid price S(t); selling revenue uses reduced price Ssell(t)=S(t)−η(t), where η reflects transaction fees, taxes or discounts.); the classifier found no reported trading result, so costs are not applicable.

Finds 3

  • Both dynamic programming and Q‑learning achieve similar approximations of the value function and yield reliable cost‑optimal decision rules. performance
    Source sentence
    A comparison of the numerical results shows that both methods can achieve similar ap-proximations of the value function and yields reliable cost-optimal decision rules.
  • Q‑learning provides a faster and more computationally efficient solution compared to dynamic programming, especially for high‑dimensional problems. performance
    Source sentence
    Although the results of Q-learning differ in some aspects, it offers a faster and computationally more efficient solution for complex control problems. This is especially useful for problems with high-dimensional state and control spaces, where the dynamic programming approach will fail due to the curse of dimensionality.
  • When extending the model to include additional state and control variables, dynamic programming becomes intractable due to the curse of dimensionality, while Q‑learning remains feasible. theoretical
    Source sentence
    By dropping the assumptions of constant mass flow ˙m and waste heat temperature T LT,in, we can extend our model and make it more general. Even though the classical backward recur-sion of dynamic programming might become intractable for this extended model due to the curse of dimensionality, we are confident that Q-learning still offers an efficient solution.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 15 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.