On-line reinforcement learning for optimization of real-life energy trading strategy

Paper: arXiv 2303.16266

Authors: Łukasz Lepak, Paweł Wawrzyński

Abstract

An increasing share of energy is produced from renewable sources by many small producers. The efficiency of those sources is volatile and, to some extent, random, exacerbating the problem of energy market balancing. In many countries, this balancing is done on the day-ahead (DA) energy markets. This paper considers automated trading on the DA energy market by a medium-sized prosumer. We model this activity as a Markov Decision Process and formalize a framework in which an applicable in real-life strategy can be optimized with off-line data. We design a trading strategy that is fed with the available environmental information that can impact future prices, including weather forecasts. We use state-of-the-art reinforcement learning (RL) algorithms to optimize this strategy. For comparison, we also synthesize simple parametric trading strategies and optimize them with an evolutionary algorithm. Results show that our RL-based strategy generates the highest market profits.

Complexity vs Empirical Score

• Math Complexity: 5.5/10
• Empirical Rigor: 6.5/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents a solid application of Markov Decision Processes and reinforcement learning algorithms to day-ahead energy trading using realistic environmental data. It offers a practical empirical comparison against evolutionary algorithm benchmarks, though reproducibility is somewhat limited by the absence of explicit open-source code links in the abstract.

Research Flowchart

  flowchart TD
    A[Goal: Optimize Real-Life Energy Trading Strategy for Prosumers] --> B[Data Inputs: Weather Forecasts & Environmental Info]
    B --> C[Model Trading Process as Markov Decision Process]
    C --> D[Optimize Trading Strategy using RL Algorithms]
    C --> E[Synthesize & Optimize Parametric Strategies via EA]
    D --> F[Evaluate Day-Ahead Market Performance]
    E --> F
    F --> G[Outcome: RL-Based Strategy Achieves Highest Market Profits]