Paper: arXiv 2501.09429
Authors: Benjamin Patrick Evans, Sihan Zeng, Sumitra Ganesh, Leo Ardon
Abstract
Agent-based models (ABMs) are valuable for modelling complex, potentially out-of-equilibria scenarios. However, ABMs have long suffered from the Lucas critique, stating that agent behaviour should adapt to environmental changes. Furthermore, the environment itself often adapts to these behavioural changes, creating a complex bi-level adaptation problem. Recent progress integrating multi-agent reinforcement learning into ABMs introduces adaptive agent behaviour, beginning to address the first part of this critique, however, the approaches are still relatively ad hoc, lacking a general formulation, and furthermore, do not tackle the second aspect of simultaneously adapting environmental level characteristics in addition to the agent behaviours. In this work, we develop a generic two-layer framework for ADaptive AGEnt based modelling (ADAGE) for addressing these problems. This framework formalises the bi-level problem as a Stackelberg game with conditional behavioural policies, providing a consolidated framework for adaptive agent-based modelling based on solving a coupled set of non-linear equations. We demonstrate how this generic approach encapsulates several common (previously viewed as distinct) ABM tasks, such as policy design, calibration, scenario generation, and robust behavioural learning under one unified framework. We provide example simulations on multiple complex economic and financial environments, showing the strength of the novel framework under these canonical settings, addressing long-standing critiques of traditional ABMs.
Complexity vs Empirical Score
- Math Complexity: 8.5/10
- Empirical Rigor: 6.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper is mathematically dense, introducing a generic framework based on Stackelberg games and POMGs with complex non-linear equations, but provides only simulated results without backtested performance metrics on real financial data.
Research Flowchart
flowchart TD
A["Research Goal: Developing a generic framework<br>for adaptive ABMs to address the Lucas critique"] --> B["Methodology: ADAGE Framework Formulation"]
B --> C["Key Component: Stackelberg Game<br>with conditional behavioral policies"]
C --> D{"Computational Process: Solve coupled non-linear equations"}
D --> E["Data/Inputs: Complex economic & financial environments"]
E --> F["Outcomes: Unified framework for<br>Policy Design, Calibration,<br>Scenario Generation, &<br>Robust Behavioral Learning"]
F --> G["Key Finding: Addresses both agent<br>and environment adaptation<br>simultaneously"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes ADAGE, a generic two‑layer Stackelberg‑game framework that jointly learns adaptive agent behavioural policies and environmental changes.
Source sentence
In this work, we develop a generic two-layer framework for ADaptive AGEnt based modelling (ADAGE) for addressing these problems. This framework formalises the bi-level problem as a Stackelberg game with conditional behavioural policies, providing a consolidated framework for adaptive agent-based modelling based on solving a coupled set of non-linear equations.
Assumes 6
- The bi‑level adaptive problem can be formalised as a Stackelberg game with a leader (outer layer) and followers (inner simulation layer). model
Source sentence
Therefore, in the following section, we formulate adaptive agent-based model design as a Stackelberg game, with the leader as the outer layer, and the followers as the (inner) agent-based simulation layer. 3 PROPOSED APPROACH We propose a generic two-layer framework, ADAGE, as visualised in fig. 1.
- Followers condition their behavioural policies on observed environment characteristics (θ̂). model
Source sentence
In the rest of the paper, we denote the behavioural policy of agent 𝑖by 𝜋𝑖(𝑎| 𝑜𝑖, ˆ𝜽𝑖) to emphasise that the followers condition their behavioural policies on the (observed) environment characteristics. When the conditioning is not explicit, it is assumed that ˆ𝜽𝑖is part of the observation 𝑜𝑖.
- The gradient domination condition holds for the agents' return functions, making every stationary point globally optimal. statistical
Source sentence
Due to the “gradient domination” condition satisfied by eq. (2), every stationary point is globally optimal [1].
- When the number of followers is large, shared policy learning can reduce the number of distinct follower policies. model
Source sentence
(4) Note that in cases of large 𝑛, in order to reduce the number of equa-tions that must be solved, shared policy learning [52] can be used for agents of similar types, reducing the set of follower behavioural policies to a subset (<< 𝑛) of shared behavioural policies.
Show 2 more
- The outer‑layer learning rate (αL) is chosen much larger than the inner‑layer rates (αF) to approximate a nested‑loop algorithm. execution
Source sentence
In general, we need to choose 𝛼𝐿,𝑡to be much larger than all 𝛼F,𝑡to approximate a nested-loop algorithm that runs multiple inner layer updates per outer layer update.
- Information processing costs for boundedly rational agents are modelled via a KL‑penalty term. model
Source sentence
While different information cost functions 𝐼can be used, here we use the KL penalty [17, 36]: 𝐼(𝜋,𝑜) = ∑︁ 𝑎 𝜋(𝑎|𝑜) log 𝜋(𝑎|𝑜) 𝜋0(𝑎) where 𝜋0(𝑎) is some prior belief, which we assume to be uniform.
Data
- TaxAI synthetic unknown
Source sentence
To demonstrate the use of ADAGE for policy design, we base an environment on TaxAI, "the most realistic eco-nomic simulator for optimal tax policy" [33], based on the canonical Bewley-Aiyagari model. Under this environment, there are multi-ple households, firms, banks, and a central government (visualised in fig. 2).
- Cobweb market game synthetic unknown
Source sentence
To demonstrate how calibration can be per-formed with ADAGE, we use the Cobweb market game [27] to explore market price fluctuations, where with human participants, we see larger fluctuations than is to be expected by perfectly ratio-nal participants, so we must calibrate the information processing costs (modulating the bounded rationality) of the agents [16]. In a cobweb market, 𝑖∈F producers must estimate the
- Financial market maker scenario synthetic unknown
Source sentence
We demonstrate robust behavioural learning under a financial environment with a market maker (MM) trading with various Liquidity Takers (LTs), where the MM may have differ-ent preferences in terms of maximising market share or profit and loss (PnL), and the goal is to learn generic MM behaviour across these preferences. There is one learning MM agent and 𝑛 zero-intelligence LT agents [18, 30] interacting in a market.
Costs
Not applicable
Finds 5
- ADAGE improves social welfare in the TaxAI policy‑design experiment, with the mean welfare increasing over training iterations. performance
Source sentence
0 200 400 600 800 1000 Training Iterations Low 8 6 4 2 0 High Social Welfare ADAGE Free Market Figure 3: Social welfare throughout training.
- Calibration of the Cobweb market using ADAGE reduces mean absolute error (MAE) to 0.005 and RMSE to 0.007, outperforming the base simulator. other
Source sentence
Table 1: Calibration metrics (based on the mean error from multiple rollouts to bootstrapped samples from experimen-tal data of [27]). Equilibria Base Simulator ADAGE MAE 0.01 0.013 0.005 RMSE 0.017 0.026 0.007 Producers receive a non-linear reward based on the accuracy of their prediction compared to the resulting market price 𝑝𝑡[27]: 𝑟F,𝑡= max(0, 1300 −260(𝑝𝑡−ˆ𝑝𝑖,𝑡)2) (12) Action The action for the calibrator is a
- In the scenario‑generation task, the ADAGE‑derived Tobin’s tax reduces market volatility in the majority of configurations (7/9 cases for std dev, 6/9 for MAPC). performance
Source sentence
The outer layer discovered automatically that scenarios with a Tobins tax successfully restricts the market volatility (reduces market fluc-tuations) in the vast majority of configurations. This reduction is observed in the standard deviation of the attendance time series (7/9 cases) and the mean absolute percentage change (6/9 cases).
- Robust behavioural learning enables a market‑maker to adapt across differing preferences without loss of performance, demonstrating algorithm‑independence. theoretical
Source sentence
This shows an additional benefit of ADAGE: when specific behaviour can be computed analytically, this can be substituted directly for any of the behavioural policies, and the remaining behaviour will be optimised w.r.t this, again demonstrating independence on the learning algorithm.
- ADAGE can substitute analytically derived policies for learned ones when available, preserving the optimisation of remaining behaviours. theoretical
Source sentence
This shows an additional benefit of ADAGE: when specific behaviour can be computed analytically, this can be substituted directly for any of the behavioural policies, and the remaining behaviour will be optimised w.r.t this, again demonstrating independence on the learning algorithm.
Checked by others
No other paper in the archive has been found to support or contradict these findings. Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (86)
- Learning and Calibrating Heterogeneous Bounded Rational Market Behaviour with Multi-Agent Reinforcement Learning (2024)
- Phantom-A RL-driven Multi-Agent Framework to Model Complex Systems (2023)
- Bounded rationality for relaxing best response and mutual consistency: the quantal hierarchy model of decision making (2024)
- A proposal for international monetary reform (1978)
- Stochastic Bilevel Optimization with Lower-Level Contextual Markov Decision Processes (2024) — link
- Bounded strategic reasoning explains crisis emergence in multi-agent market games (2023)
- Bayesian Optimization : Open source constrained global optimization tool for Python (2014–) — link
- Survey: Development and analysis of a games-based crisis scenario generation system (2016)
and 78 more
- Effect of Tobin Tax on Trading Decisions in an Experimental Minority Game (2019) — link
- Getting at systemic risk via an agent-based model of the housing market (2012)
- Effects of Tobin taxes in minority game markets (2009)
- Population Synthesis as Scenario Generation for Simulation-based Planning under Uncertainty (2024)
- ABIDES-Economist: Agent-Based Simulation of Economic Systems with Learning Agents (2024) — link
- AgentTorch: Agent-based Modeling with Automatic Differentiation (2023)
- Letter-value plots: Boxplots for large data (2017)
- Evaluating the stability of non-adaptive trading in continuous double auctions (2018)
- Proximal policy optimization algorithms (2017) — link
- The predictive power of zero intelligence in financial markets (2005)
- Parameterised response zero intelligence traders (2023)
- Zero intelligence in economics and finance (2012)
- Agent-based modeling in economics and finance: Past, present, and future (2022)
- Foundations of complexity economics (2021)
- Do differentiable simulators give better policy gradients? (2022)
- Behavioral game theory: Experiments in strategic interaction (2011)
- TaxAI: A Dynamic Economic Simulator and Benchmark for Multi-agent Reinforcement Learning (2024)
- Differentiable Agent-based Epidemiology (2023)
- Challenges, tasks, and opportunities in modeling agent-based complex systems (2021)
- Optimal auctions through deep learning: Advances in differentiable economics (2024)
- On learning agent-based models from data (2023)
- Differentiable economics for randomized affine maximizer auctions (2023)
- Optimal Automated Market Makers: Differentiable Economics and Strong Duality — → 2402.09129
- Gradient-assisted calibration for financial agent-based models (2023)
- The AI Economist: Taxation policy design via two-level deep multiagent reinforcement learning (2022)
- Calibration of shared equilibria in general sum partially observable Markov games (2020)
- Meta-RL for Multi-Agent RL: Learning to Adapt to Evolving Agents (2022)
- Towards multi-agent reinforcement learning-driven over-the-counter market simulations (2023)
- The economy needs agent-based modelling (2009)
- Stackelberg security games: Looking beyond a decade of success (2018)
- Fast two-time-scale stochastic gradient method with applications in reinforcement learning (2024)
- Learning in speculative bubbles: Theory and experiment (2021)
- Towards a science of security games (2016)
- oTree: The bubble game (2019)
- The bubble game: An experimental study of speculation (2013)
- The bubble game: A classroom experiment (2016)
- Building a foundation for data-driven, interpretable, and robust policy design using the ai economist (2021) — link
- Modelling Agent Decision Making in Agent-based Simulation-Analysis Using an Economic Technology Uptake Model (2023)
- Econometric policy evaluation: A critique (1976)
- Stackelberg POMDP: A Reinforcement Learning Approach for Economic Design (2022) — link
- Computing optimal equilibria and mechanisms via learning in zero-sum extensive-form games (2024)
- Control of parametric games (2019)
- Repeated Stackelberg security games: Learning with incomplete state information (2020)
- Canvas: A canadian behavioral agent-based model (2022)
- Models of financial stability and their application in stress tests (2018)
- Forecasting in a complex environment: Machine learning sales expectations in a Stock Flow Consistent Agent-Based simulation model (2022)
- Learning in cobweb experiments (2007)
- Optimal tax progressivity: An analytical framework (2017)
- Tax and education policy in a heterogeneous-agent economy: What levels of redistribution maximize growth and efficiency? (2002)
- Generative agents: Interactive simulacra of human behavior (2023)
- Modeling expectations in agent-based models—An application to central bank’s communication and monetary policy (2015)
- A short walk on the wild side: Agent-based models and their implications for macroeconomic analysis (2018)
- Reinforcement Learning for Economic Policy: A New Frontier? (2022) — link
- Stackelberg actor-critic: Game-theoretic reinforcement learning algorithms (2022)
- A comparison of economic agent-based model calibration methods (2020)
- Don’t Simulate Twice: One-Shot Sensitivity Analyses via Automatic Differentiation (2023)
- RLlib: Abstractions for distributed reinforcement learning (2018)
- Automated mechanism design: A new application area for search algorithms (2003)
- Implicit Learning Dynamics in Stackelberg Games: Equilibria Characterization, Convergence Analysis, and Empirical Study (2020) — link
- Learning stackelberg equilibria and applications to economic design games (2022)
- Oracles & followers: Stackelberg equilibria in deep multi-agent reinforcement learning (2023)
- Using machine learning for agent specifications in agent-based models and simulations: A critical review and guidelines (2023)
- Quantal response equilibria for normal form games (1995)
- Thermodynamics as a theory of decision-making with information-processing costs (2013)
- Position: Social Environment Design Should be Further Developed for AI -based Policy-Making (2024)
- Segregation dynamics with reinforcement learning and agent based modeling (2020)
- Deep reinforcement learning in agent-based simulations for optimal media planning (2023)
- MADES: A Unified Framework for Integrating Agent-Based Simulation with Multi-Agent Reinforcement Learning (2021) — link
- Towards an multilevel agent-based model for traffic simulation (2017)
- Multi-level agent-based simulations: Four design patterns (2018)
- Levelspace: A netlogo extension for multi-level agent-based modeling (2020)
- Multilevel agent-based modeling of system of systems (2015)
- " A Game of Thrones" When Human Behavior Models Compete in Repeated Stackelberg Security Games (2015)
- Agent-based models: understanding the economy from the bottom up (2016)
- MAML and ANIL provably learn representations (2022)
- On the theory of policy gradient methods: Optimality, approximation, and distribution shift (2021)
- A two-time-scale stochastic optimization framework with applications in control and reinforcement learning (2024)
- A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic (2023)
1 of 86 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.