{"paper_id":"2307.01085","title":"Some challenges of calibrating differentiable agent-based models","authors":["Arnau Quera-Bofarull","Joel Dyer","Anisoara Calinescu","Michael Wooldridge"],"paper_date":"2023-07-03","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2307.01085v1","page_url":"https://thequant.space/flowcharts/some-challenges-of-calibrating-differentiable-agent-based-mo/","scores":{"empirical_rigor":6,"math_complexity":8.5,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes a differentiating strategy that alternates between forward-mode AD (FMAD) and reverse-mode AD (RMAD) when calibrating agent-based models.","quote":"To address this, in subsection 3.2 we discuss a differentiating strategy that alternates between FMAD and RMAD when calibrating ABMs, and we apply it to an epidemiological simulation involving 8 million agents. 2.3.","summary":"Agent-based models (ABMs) are a promising approach to modelling and reasoning about complex systems, yet their application in practice is impeded by their complexity, discrete nature, and the difficulty of performing parameter inference and optimisation tasks. This in turn has sparked interest in…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The loss function ℓ(y,θ) is chosen to be a differentiable function of the model parameters θ.","type":"model","stated":"explicit","quote":"(6) Further details are provided in Appendix B. Thus, provided ℓis chosen to be a differentiable function of θ, this enables us to employ gradient-based approaches to minimising the objective (5) that exploit the reparameterisation trick.","support":"full"},{"assumption":"Synthetic data y are generated from the Brock & Hommes model at known parameter values for calibration experiments.","type":"data","stated":"explicit","quote":"Fixing g1 = b1 = b4 = 0 and g4 = 1.01, we consider the task of calibrating parameters g2, g3, b2, b3 given syn-thetic data y = (y1, . . . , yT ) generated from the model at (g2, g3, b2, b3) = (0.9, 0.9, 0.2, −0.2) with T = 100.","support":"full"},{"assumption":"The maximum mean discrepancy (MMD) loss used for calibration is differentiable and deterministic when a Gaussian kernel is employed.","type":"model","stated":"explicit","quote":"We follow Cherief-Abdellatif & Alquier (2020) and target the generalised posterior (4) given by the choice ℓ(y, θ) = MMD2(PT , Pθ), (7) where MMD2(PT , Pθ) is the maximum mean discrep-ancy between the empirical measure of returns PT = (y1, . . . , yT ) and the distribution Pθ of returns implied by the simulator at parameters θ.","support":"full"}]},"datasets":{"status":"available","items":[{"name":"Synthetic data from the Brock & Hommes model","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"Fixing g1 = b1 = b4 = 0 and g4 = 1.01, we consider the task of calibrating parameters g2, g3, b2, b3 given syn-thetic data y = (y1, . . . , yT ) generated from the model at (g2, g3, b2, b3) = (0.9, 0.9, 0.2, −0.2) with T = 100.","support":"full"},{"name":"Synthetic time-series of daily infections for the JUNE model","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"We generate a synthetic time-series of daily infec-tions for 50 days using some assumed parameters that we aim to recover through our calibration process.","support":"full"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"synthetic"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"none","findings":[{"finding":"Using a finite gradient horizon (H=0) dramatically improves the gradient‑assisted training of the variational posterior for the Brock & Hommes model.","kind":"performance","quote":"In this experiment, the best performance was observed while using a gradient horizon of H = 0.","support":"full"},{"finding":"With the vanilla pathwise derivative, the objective decreases slowly and offers no improvement over the score‑based gradient.","kind":"negative","quote":"This can be seen in Figure 1, in which the objective function decreases slowly with the number of epochs when trained with AdamW (Loshchilov & Hutter, 2017) and using the vanilla pathwise derivative (red curve).","support":"full"},{"finding":"The GVI calibration of the JUNE model converges after approximately 3,000 model evaluations, demonstrating simulation‑efficient calibration with gradient‑assisted methods.","kind":"performance","quote":"The flow converges after approxi-mately 3,000 model evaluations, highlighting the potential for simulation-efficient calibration with gradient-assisted methods.","support":"full"}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":9,"citations":{"cited_by_in_corpus":3,"by_independent_authors":3,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":1,"unclassified":2},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":true,"code_url":"https://github.com/arnauqb/blackbirds","note":null},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":2,"extracted_at":"2026-11-14T20:02:13Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}