Paper: arXiv 2301.08360

Authors: Yuanrong Wang, Vignesh Raja Swaminathan, Nikita P. Granger, Carlos Ros Perez, Christian Michler

Abstract

In this paper, we discuss the Dutch power market, which is comprised of a day-ahead market and an intraday balancing market that operates like an auction. Due to fluctuations in power supply and demand, there is often an imbalance that leads to different prices in the two markets, providing an opportunity for arbitrage. To address this issue, we restructure the problem and propose a collaborative dual-agent reinforcement learning approach for this bi-level simulation and optimization of European power arbitrage trading. We also introduce two new implementations designed to incorporate domain-specific knowledge by imitating the trading behaviours of power traders. By utilizing reward engineering to imitate domain expertise, we are able to reform the reward system for the RL agent, which improves convergence during training and enhances overall performance. Additionally, the tranching of orders increases bidding success rates and significantly boosts profit and loss (P&L). Our study demonstrates that by leveraging domain expertise in a general learning problem, the performance can be improved substantially, and the final integrated approach leads to a three-fold improvement in cumulative P&L compared to the original agent. Furthermore, our methodology outperforms the highest benchmark policy by around 50% while maintaining efficient computational performance.

Complexity vs Empirical Score

  • Math Complexity: 6.5/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents an innovative multi-agent DRL framework incorporating domain-specific reward engineering and order tranching for European power market arbitrage. It shows strong empirical results with a significant P&L improvement over standard benchmarks, though reproducible code details could be expanded.

Research Flowchart

  flowchart TD
    A["Research Goal: Optimize European Power Arbitrage<br><i>(Day-ahead vs. Intraday Balancing Markets)</i>"] --> B["Data Inputs: Dutch Power Market Auction & Price Data"]
    
    B --> C["Collaborative Dual-Agent DRL Framework<br><i>(Bi-level Simulation & Optimization)</i>"]
    
    C --> D["Domain Adaptation & Imitation Learning"]
    
    D --> D1["Domain-Guided Reward Engineering<br><i>(Imitates expert traders for faster convergence)</i>"]
    D --> D2["Order Tranching Strategy<br><i>(Improves bidding success rates)</i>"]
    
    D1 --> E["Final Integrated Trading Policy"]
    D2 --> E
    
    E --> F["Key Outcomes:<br>• 3x P&L improvement over baseline RL<br>• Outperforms top benchmark by ~50%<br>• High computational efficiency"]