Paper: arXiv 2610.26789

Authors: Skander Lejmi

Abstract

A market maker provides liquidity by quoting prices to buy and sell an asset, earning the spread while managing inventory risk. We study market makers trained with multi-agent reinforcement learning on prediction markets, specifically Polymarket, whose thin order books pose distinct challenges. Three challenges underpin this research: adverse selection may cause a policy trained on real data to stop quoting; the consequences of a quote can arrive many steps after the action; and prediction-market L2 data does not identify individual orders needed for simulation. We address these challenges by reconstructing order-level LOBSTER-format data from Polymarket and training agents in JaxMARL-HFT on 417 real markets. We enrich the baseline market maker’s observation space with the book and recent flow, redesign its actions and inventory controls, and propose a cooperative multi-agent decomposition into a quoter and a risk manager trained using recurrent Independent Proximal Policy Optimisation (IPPO) with RNN policies. We investigate reward shaping, particularly reward blending and post-fill markout, to mitigate credit-assignment difficulties between the agents. Our redesigned single-agent market maker quotes more frequently and incurs smaller held-out losses than the retrained JaxMARL-HFT baseline, although the contributions of jointly changed observations, actions and risk constraints are not separately identified. Decomposition alone performs poorly; careful reward design substantially improves the desk relative to the unshaped split and yields our best result. Models are evaluated on realised end-of-episode portfolio value on held-out markets. None establishes positive average portfolio value in periodic held-out evaluation, and each configuration was trained with one seed. The environment, data and protocol lay a foundation for makers that quote many linked markets at once.

Complexity vs Empirical Score

  • Math Complexity: 6.0/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents a novel application of multi-agent reinforcement learning to market making in prediction markets, addressing specific challenges like thin order books and adverse selection. It demonstrates strong empirical rigor through data reconstruction, training on real market data, and evaluating models on held-out sets, despite the negative average portfolio value. The methodology is well-articulated, and the contributions are clearly outlined.

Research Flowchart

  flowchart TD
    A[Research Goal: Develop MARL Market Makers for Prediction Markets] --> B{Key Challenges Addressed};
    B --> C1[Adverse Selection];
    B --> C2[Delayed Consequences of Quotes];
    B --> C3[Lack of Order-Level L2 Data];
    C3 --> D[Data Reconstruction: Polymarket L2 to LOBSTER Format];
    D --> E[Methodology: MARL Training on 417 Real Markets];
    E --> F{Agent Design & Training};
    F --> F1[Observation Space Enrichment (Book & Flow)];
    F --> F2[Action & Inventory Control Redesign];
    F --> F3[Cooperative Multi-Agent Decomposition (Quoter & Risk Manager)];
    F --> F4[Recurrent IPPO with RNN Policies];
    F --> F5[Reward Shaping (Blending & Post-Fill Markout)];
    E --> G[Evaluation: Realized End-of-Episode Portfolio Value on Held-Out Markets];
    G --> H{Key Findings/Outcomes};
    H --> H1[Redesigned Single-Agent: More Quotes, Smaller Held-Out Losses than Baseline];
    H --> H2[Decomposition Alone: Poor Performance];
    H --> H3[Decomposition + Reward Shaping: Substantial Improvement, Best Result];
    H --> H4[No Configuration Achieved Positive Average Portfolio Value];
    H --> H5[Foundation for Makers on Linked Markets Laid];