{"paper_id":"2610.01348","title":"Verify Claims, Not Scores: Evidence-Based Verification of Modular Agents","authors":["Ali Atiah Alzahrani"],"paper_date":"2026-10-01","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2610.01348","page_url":"https://thequant.space/flowcharts/verify-claims-not-scores-evidence-based-verification-of-modular-agents/","scores":{"empirical_rigor":8,"math_complexity":6.5,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes a claim‑specific verification audit for modular agents that records evidence for each claim instead of scoring the agent.","quote":"We introduce a claim-specific verification audit for modular agents that plan, act, check and refine.","summary":"When developers change one component of an agent, such as its controller, a learned model or its verifier, they usually judge the change by an aggregate task score. That score cannot tell whether improvement was attainable, which component lost value, or what the agent's own checks certify. We…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The environment is a synthetic market with recorded hidden regimes.","type":"market","stated":"explicit","quote":"work in a controlled setting that provides all three: a synthetic market with recorded hidden regimes"},{"assumption":"Portfolio weights are long‑only, fully invested, lie in a per‑asset box [0,0.25] and turnover is limited to 0.20 in ℓ1 norm per rebalance.","type":"execution","stated":"explicit","quote":"At each rebalance t the agent chooses weights wt that are long‑only and fully invested (the budget), lie in a per‑asset box [0, 0.25], and move at most 0.20 in ℓ1 norm from wt−1 (the turnover cap)"},{"assumption":"Estimation is strict, walk‑forward, purged and embargoed.","type":"statistical","stated":"explicit","quote":"Estimation is strict, purged, and embargoed walk‑forward"},{"assumption":"Portfolio loss is uniformly bounded (bounded loss assumption for the certificate).","type":"statistical","stated":"explicit","quote":"The portfolio loss −w⊤r is bounded by M uniformly over the feasible set"},{"assumption":"The calibration score sequence is approximately locally exchangeable.","type":"statistical","stated":"explicit","quote":"The score sequence is approximately locally exchangeable in the sense adaptive conformal inference is designed to tolerate"}]},"datasets":{"status":"available","items":[{"name":"synthetic market with recorded hidden regimes","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"a synthetic market with recorded hidden regimes, in which an agent allocates a portfolio under explicit constraints"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"synthetic"},"transaction_costs":{"status":"available","modeled":"yes","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R1","candidates":{"front_matter":"yes","evidence_card":"unclear","evidence_card_quoted":false},"meaning":"costs are part of the reported headline results"},"results":{"status":"available","evaluation_type":"trading-pnl","findings":[{"finding":"The value of perfect regime information depends on the action set used to measure it.","kind":"performance","quote":"the audit shows that the value of perfect regime information depends on the action set used to measure it"},{"finding":"The scenario generator discards most of the regime signal while better local fidelity does not improve decisions.","kind":"negative","quote":"the scenario generator discards most of the regime signal while better local fidelity does not improve decisions"},{"finding":"Improved local scores do not establish downstream improvement.","kind":"negative","quote":"Improved local scores do not establish downstream improvement"},{"finding":"The runtime verifier can be bypassed with no visible change in outcomes.","kind":"negative","quote":"the runtime verifier can be bypassed with no visible change in outcomes"},{"finding":"Headroom (potential improvement) is a property of the action representation, not of information alone.","kind":"theoretical","quote":"Measured headroom depends on the action representation, not on the information alone"}],"reported_performance":null,"out_of_sample":{"method":"walk-forward","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":24,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:03:42Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}