Paper: arXiv 2303.07393

Authors: Matthew Dicks, Andrew Paskaramoorthy, Tim Gebbie

Abstract

We consider the dynamics and the interactions of multiple reinforcement learning optimal execution trading agents interacting with a reactive Agent-Based Model (ABM) of a financial market in event time. The model represents a market ecology with 3-trophic levels represented by: optimal execution learning agents, minimally intelligent liquidity takers, and fast electronic liquidity providers. The optimal execution agent classes include buying and selling agents that can either use a combination of limit orders and market orders, or only trade using market orders. The reward function explicitly balances trade execution slippage against the penalty of not executing the order timeously. This work demonstrates how multiple competing learning agents impact a minimally intelligent market simulation as functions of the number of agents, the size of agents’ initial orders, and the state spaces used for learning. We use phase space plots to examine the dynamics of the ABM, when various specifications of learning agents are included. Further, we examine whether the inclusion of optimal execution agents that can learn is able to produce dynamics with the same complexity as empirical data. We find that the inclusion of optimal execution agents changes the stylised facts produced by ABM to conform more with empirical data, and are a necessary inclusion for ABMs investigating market micro-structure. However, including execution agents to chartist-fundamentalist-noise ABMs is insufficient to recover the complexity observed in empirical data.

Complexity vs Empirical Score

  • Math Complexity: 6.5/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: This paper presents a novel multi-agent ecosystem integrating reinforcement learning optimal execution agents within a 3-trophic agent-based market model. It effectively demonstrates that adding learning agents aligns simulated stylized facts more closely with empirical market microstructure, though empirical testing relies primarily on stylized facts rather than direct historical tick backtesting.

Research Flowchart

  flowchart TD
    A["Research Goal: Impact of RL Trading Agents on Market ABM Dynamics & Empirical Realism"] --> B["Market Ecology Setup: 3-Trophic ABM<br>(RL Optimal Execution Agents, Liquidity Takers, Electronic Liquidity Providers)"]
    B --> C["Configure RL Agents & Reward Function<br>(Varied State Spaces, Order Sizes, Action Types: LO/MO vs MO-only)"]
    C --> D["Simulate Market Dynamics in Event Time & Generate Phase Space Plots"]
    D --> E["Evaluate Outcomes vs Empirical Market Stylized Facts"]
    E --> F1["Finding 1: RL agents alter ABM dynamics to align closer with empirical data"]
    E --> F2["Finding 2: Adding execution agents alone is insufficient to fully recover empirical complexity"]