Paper: arXiv 2309.01784

Authors: Song Wei, Andrea Coletta, Svitlana Vyetrenko, Tucker Balch

Abstract

In many applications involving multi-agent system (MAS), it is imperative to test an experimental (Exp) autonomous agent in a high-fidelity simulator prior to its deployment to production, to avoid unexpected losses in the real-world. Such a simulator acts as the environmental background (BG) agent(s), called agent-based simulator (ABS), aiming to replicate the complex real MAS. However, developing realistic ABS remains challenging, mainly due to the sequential and dynamic nature of such systems. To fill this gap, we propose a metric to distinguish between real and synthetic multi-agent systems, which is evaluated through the live interaction between the Exp and BG agents to explicitly account for the systems’ sequential nature. Specifically, we characterize the system/environment by studying the effect of a sequence of BG agents’ responses to the environment state evolution and take such effects’ differences as MAS distance metric; The effect estimation is cast as a causal inference problem since the environment evolution is confounded with the previous environment state. Importantly, we propose the Interactive Agent-Guided Simulation (INTAGS) framework to build a realistic ABS by optimizing over this novel metric. To adapt to any environment with interactive sequential decision making agents, INTAGS formulates the simulator as a stochastic policy in reinforcement learning. Moreover, INTAGS utilizes the policy gradient update to bypass differentiating the proposed metric such that it can support non-differentiable operations of multi-agent environments. Through extensive experiments, we demonstrate the effectiveness of INTAGS on an equity stock market simulation example. We show that using INTAGS to calibrate the simulator can generate more realistic market data compared to the state-of-the-art conditional Wasserstein Generative Adversarial Network approach.

Complexity vs Empirical Score

  • Math Complexity: 8.0/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced mathematics including causal inference, reinforcement learning formulation with policy gradients, and stochastic policy optimization, indicating high complexity. It includes extensive experiments on equity stock market simulation, comparing results to state-of-the-art methods, demonstrating substantial empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal"] --> B["Propose MAS Distance Metric"]
  B --> C["Estimate Effects via Causal Inference"]
  C --> D["INTAGS Framework: RL Policy Optimization"]
  D --> E["Policy Gradient Update"]
  E --> F["Calibrate ABS"]
  F --> G["Key Findings: Realistic Equity Simulation"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes the Interactive Agent-Guided Simulation (INTAGS) framework for building realistic agent-based simulators by optimizing a novel multi-agent system distance metric, and its market application called Algorithmic Trading‑guided Market Simulator (ATMS).

Source sentence
Importantly, we propose the Interactive Agent-Guided Simulation (INTAGS) framework to build a realistic ABS by optimizing over this novel metric. Based on this novel metric, we develop an online training framework, named Interactive Agent-Guided Simulation (INTAGS).

Assumes 5

  • The simulator can be represented as a stochastic policy optimized via policy gradient without differentiating the metric. model
    Source sentence
    To adapt to any environment with interactive sequential decision making agents, INTAGS formulates the simulator as a stochastic policy in reinforcement learning. Moreover, INTAGS utilizes the policy gradient update to bypass differentiating the proposed metric such that it can support non-differentiable operations of multi-agent environments.
  • The real environment is approximated by the ABIDES parametric simulator, treated as ground truth. market
    Source sentence
    Since we do not have access to real trading data, we consider a state-of-art parametric simulator as the real Env, where it is easier to obtain the “ground truth”; In particular, we take ABIDES [Byrd et al., 2020] with explicitly specified/parametric BG agents as “reality”.
  • The experimental agent interacts with the environment only through offline replay data, which does not interfere with production. data
    Source sentence
    INTAGS only requires offline inter-actions/replays of the real-world experimental agent with the real environment, and thus will not interfere with the real-world production system; The framework is online because the objective function, or rather the proposed distance metric, is obtained through the online/live interaction of the real-world experimental agent with the synthetic environment.
  • The optimal execution task assumes a fixed share size and time horizon for the AT agent. execution
    Source sentence
    The Exp agent is chosen as the AT agent that performs the optimal execution task, aiming to acquire a fixed share of a single asset within a fixed time horizon while minimizing the transaction cost.
Show 1 more
  • Feedback function f can be estimated from a small number of rollouts (e.g., Mkt2NextReturn). statistical
    Source sentence
    Thus, we choose Mkt2NextReturn as f that can differentiate markets with a small number of rollouts.

Data

  • ABIDES simulated market data synthetic public
    Source sentence
    Since we do not have access to real trading data, we consider a state-of-art parametric simulator as the real Env, where it is easier to obtain the “ground truth”; In particular, we take ABIDES [Byrd et al., 2020] with explicitly specified/parametric BG agents as “reality”. Using ABIDES data we train our simulated world BG agent.
  • Rollout data T collected from experimental agent interactions synthetic unknown
    Source sentence
    We use an ordered list T to denote the collection of states and actions when the Exp agent π interacts with the Env, and we name the collected data during (or after) this 6 interaction as rollout.

Recognised sources: ABIDES simulator

Costs

Not applicable The paper reports no trading or execution result; the cost the extractor found is a model parameter.

The paper-text extraction found a cost parameter inside a model (The AT agent’s objective is to minimize transaction cost while acquiring a fixed share amount within a time horizon.); the classifier found no reported trading result, so costs are not applicable.

Finds 5

  • INTAGS calibrated simulators generate market data that is more realistic than the state‑of‑the‑art conditional Wasserstein GAN approach. performance
    Source sentence
    We show that using INTAGS to calibrate the simulator can generate more realistic market data compared to the state-of-the-art conditional Wasserstein Generative Adversarial Network approach.
  • ATMS training reduces the AT‑based metric from 0.7156 to 0.6176, indicating improved similarity to reality. performance
    Source sentence
    The corresponding AT-based metric is optimized from 0.7156 to 0.6176.
  • During ATMS training, volume at the first 10 LOB levels becomes increasingly similar to the real environment while the AT‑based metric is minimized. performance
    Source sentence
    We can observe that, during the ATMS training, volume at first 10-levels is “increasingly similar” (in terms of magnitude and matching demand-supply) to reality while our AT-based metric is minimized. The ATMS can generate market data that is “increasingly” similar to reality while minimizing our AT-based metric.
  • ATMS outperforms the cWGAN baseline in matching stylized facts such as volume distribution and buy/sell balance. performance
    Source sentence
    Next, we show that our ATMS outperforms existing cWGAN baseline [Coletta et al., 2022]; See Appendix C.3 for further details. Interestingly, we find that the stylized facts are not only similar to the target market but also show more balanced BUY and SELL volumes.
  • INTAGS training can be completed in about 2 hours on AWS r6i.24xlarge (96 CPUs, 768 GiB) by tuning hyperparameters. other
    Source sentence
    This is because T0 (and b) determines how many BG agent actions are sampled and penalized, and thus can be treated as batch-size in stochastic GD; This makes T0 a tunable hyperparame-ter; By carefully tuning T0 (and b; see the hyperparameter selection in Appendix D.1), we can finish the INTAGS training within a reasonable time frame (around 2 hours) using Amazon Web Services (r6i.24xlarge, 96 CPUs, 768 GiB memory).

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 29 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.