Paper: arXiv 2308.01910

Authors: Jonas Hanetho

Abstract

The energy transition has increased the reliance on intermittent energy sources, destabilizing energy markets and causing unprecedented volatility, culminating in the global energy crisis of 2021. In addition to harming producers and consumers, volatile energy markets may jeopardize vital decarbonization efforts. Traders play an important role in stabilizing markets by providing liquidity and reducing volatility. Several mathematical and statistical models have been proposed for forecasting future returns. However, developing such models is non-trivial due to financial markets’ low signal-to-noise ratios and nonstationary dynamics. This thesis investigates the effectiveness of deep reinforcement learning methods in commodities trading. It formalizes the commodities trading problem as a continuing discrete-time stochastic dynamical system. This system employs a novel time-discretization scheme that is reactive and adaptive to market volatility, providing better statistical properties for the sub-sampled financial time series. Two policy gradient algorithms, an actor-based and an actor-critic-based, are proposed for optimizing a transaction-cost- and risk-sensitive trading agent. The agent maps historical price observations to market positions through parametric function approximators utilizing deep neural network architectures, specifically CNNs and LSTMs. On average, the deep reinforcement learning models produce an 83 percent higher Sharpe ratio than the buy-and-hold baseline when backtested on front-month natural gas futures from 2017 to 2022. The backtests demonstrate that the risk tolerance of the deep reinforcement learning agents can be adjusted using a risk-sensitivity term. The actor-based policy gradient algorithm performs significantly better than the actor-critic-based algorithm, and the CNN-based models perform slightly better than those based on the LSTM.

Complexity vs Empirical Score

  • Math Complexity: 8.0/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper uses advanced deep reinforcement learning mathematics (policy gradients, CNNs, LSTMs, stochastic dynamical systems) and provides robust empirical results with specific backtests, hyperparameters, and performance metrics on real commodity data.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Assess Deep RL for<br>Commodity Trading"] --> B["Methodology:<br>Formalize as Stochastic System<br>with Adaptive Time-Discretization"]
  B --> C["Data:<br>Front-Month Natural Gas Futures<br>(2017–2022)"]
  C --> D["Computations:<br>Train CNN/LSTM Agents<br>via Policy Gradient (Actor & Actor-Critic)"]
  D --> E["Findings:<br>83% Higher Sharpe Ratio vs. Buy-Hold<br>Actor-Only > Actor-Critic<br>CNNs > LSTMs"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes two policy‑gradient algorithms (an actor‑based and an actor‑critic‑based) and a novel reactive, adaptive time‑discretization scheme for commodity trading.

Source sentence
Two policy gradient algorithms, an actor-based and an actor-critic-based, are proposed to optimize a transaction-cost- and risk-sensitive agent.

Assumes 3

  • No slippage: sufficient liquidity to fill any order at the quoted price. execution
    Source sentence
    No slippage, i.e., there is sufficient liquidity in the market to fill any orders placed by the agent, regardless of size, at the quoted price.
  • No market impact: the agent’s trades are too small to move the market. market
    Source sentence
    No market impact, i.e., the money invested by the agent is not significant enough to move the market.
  • Fractional trading assumption: positions can be scaled arbitrarily, which is less reasonable with small capital. other
    Source sentence
    The fractional trading assumption is less reasonable the smaller the amount of capital available to the agent.

Data

  • front‑month TTF Natural Gas Futures contracts market-prices unknown
    2011-2022 · Natural Gas futures
    Source sentence
    The dataset consists of the front‑month TTF Natural Gas Futures contracts from 2011 to 2022.

Costs

Modeled Costs are part of the reported headline results.

A transaction‑cost fraction λc is included in the reward function and set to a reasonable value reflecting market conditions.

Source sentence
∈[0,1] is the transaction cost fraction that is assumed to be identical for buying and selling.

Finds 3

  • The deep RL agents achieve an 83 % higher Sharpe ratio than the buy‑and‑hold baseline on out‑of‑sample backtests. performance
    Source sentence
    On average, the deep reinforcement learning agents produce an 83% higher Sharpe ratio out‑of‑sample than the buy‑and‑hold baseline.
  • The actor‑based policy gradient algorithm yields about 49 % higher Sharpe on average compared to the deterministic actor‑critic algorithm. performance
    Source sentence
    algorithm produces significantly higher Sharpe (49% on average) than the deterministic actor‑critic algorithm, suggesting that an actor‑based policy gradient method is more suited to algorithmic trading in an online, continuous time setting.
  • CNN‑based models obtain roughly 5 % higher Sharpe than LSTM‑based models, possibly due to vanishing‑gradient issues in LSTMs. performance
    Source sentence
    The parametric function approximator based on the CNN architecture performs slightly better (5% higher Sharpe on average) than the LSTM, possibly due to the problem of vanishing gradients for the LSTM.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 20 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.