Paper: arXiv 2307.11685

Authors: Chuheng Zhang, Yitong Duan, Xiaoyu Chen, Jianyu Chen, Jian Li, Li Zhao

Abstract

Optimized trade execution is to sell (or buy) a given amount of assets in a given time with the lowest possible trading cost. Recently, reinforcement learning (RL) has been applied to optimized trade execution to learn smarter policies from market data. However, we find that many existing RL methods exhibit considerable overfitting which prevents them from real deployment. In this paper, we provide an extensive study on the overfitting problem in optimized trade execution. First, we model the optimized trade execution as offline RL with dynamic context (ORDC), where the context represents market variables that cannot be influenced by the trading policy and are collected in an offline manner. Under this framework, we derive the generalization bound and find that the overfitting issue is caused by large context space and limited context samples in the offline setting. Accordingly, we propose to learn compact representations for context to address the overfitting problem, either by leveraging prior knowledge or in an end-to-end manner. To evaluate our algorithms, we also implement a carefully designed simulator based on historical limit order book (LOB) data to provide a high-fidelity benchmark for different algorithms. Our experiments on the high-fidelity simulator demonstrate that our algorithms can effectively alleviate overfitting and achieve better performance.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents a theoretical framework (ORDC) and derives generalization bounds, indicating high mathematical complexity, while also implementing a high-fidelity simulator and evaluating algorithms, demonstrating strong empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Address Overfitting in<br>RL for Trade Execution"] --> B["Methodology<br>Model as Offline RL with<br>Dynamic Context (ORDC)"]
  B --> C["Input Data<br>Historical Limit Order Book<br>LOB Data"]
  C --> D["Computational Process<br>Learn Compact Context<br>Representations"]
  D --> E{"Key Finding 1<br>Generalization Bound<br>Limited Samples cause Overfitting"}
  D --> F{"Key Finding 2<br>Proposed Algorithms<br>Effective Overfitting Mitigation"}
  E --> G["Outcome<br>High-Fidelity Simulator<br>Benchmark"]
  F --> G

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes the Offline RL with Dynamic Context (ORDC) framework and two algorithms, CASH and CATE, to learn compact, generalizable context representations for trade execution.

Source sentence
We propose two algorithms: CASH (Context Aggregate with Hand-crafted Statistics) which can learn interpretable models, and CATE (Context Aggregate with End-to-end Training) which does not require domain knowledge.

Assumes 5

  • The immediate reward is bounded between 0 and 1 and the state and action spaces are discrete. statistical
    Source sentence
    Assumption 1 (Regularity). C, X, S, and A are discrete and the immediate reward r(x, s, a) ∈[0, 1], ∀x, s, a.
  • The transition dynamics of the private state given context and action (Ps) are known. model
    Source sentence
    Moreover, Ps (e.g., the rules to match the orders) is usually simple, whereas Pc
  • The market variable (context) evolves independently of the agent’s actions and state. market
    Source sentence
    The context evolves independently and is not affected by the action or the state.
  • The context space is high‑dimensional but contains a low‑dimensional latent representation sufficient for decision making. model
    Source sentence
    The context c ∈C is usually high-dimensional and corresponds to a more compact latent context x ∈X (e.g., the key information for making trading decisions).
Show 1 more
  • The offline dataset contains limited context sequences for training. data
    Source sentence
    Since it is usually hard to increase the number of sampled context sequences for training in practice, we find another way to address the overfitting problem.

Data

  • high‑fidelity trade execution simulator synthetic unknown
    Source sentence
    In the experiment, we first implement a high‑fidelity and open‑source simulator for trade execution to provide a uniform backtest for different methods.
  • simplified trade execution task data synthetic unknown
    Source sentence
    We design a simplified trade execution task that motivates us to learn a compact

Costs

Modeled Costs are part of the reported headline results.

Trading cost is captured in the reward term r1 = n_t * \bar{p}_t (negative trading cost) plus regularization terms.

Source sentence
In our algorithms, the reward function consists of a revenue term r1 and a regularization term r2, i.e., r_t = r1+βr2 where

Finds 3

  • The proposed algorithms (CASH and CATE) achieve higher evaluation rewards (lower trading costs) than baseline methods, demonstrating better generalization. performance
    Source sentence
    With this simulator, we find that previous state-of-the-art algorithms suffer from over‑fitting and our algorithm outperforms these baselines due to better generalization.
  • CASH achieves an evaluation reward of 1.8028 (train) vs 1.8016 (train) with low gap, outperforming the Base model especially at lower data volumes. performance
    Source sentence
    CASH 1.8016 0.0246 1.8028 0.0187 -0.0013 0.0321
  • CATE also outperforms the Base model and shows improved stability across different data volumes. performance
    Source sentence
    CATE 1.7143 0.0281 1.7004 0.0265 0.0139 0.0252

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 40 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 5 papers in the archive (4 by independent authors): MacroHFT: Memory Augmented Context-aware Reinforcement … (discusses its findings)

Code

https://github.com/zhangchuheng123/RL4Execution — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.