Paper: arXiv 2605.03310
Authors: Maksym Nechepurenko, Pavel Shuvalov
Abstract
Multi-agent LLM systems fail in production at rates between 41% and 87%, mostly due to coordination defects rather than base-model capability. Existing responses split between cataloguing failure modes empirically and shipping declarative orchestration frameworks as engineering tools; neither delivers a principled mapping from coordination configuration to predictable failure-mode signature. We argue that coordination should be treated as a configurable architectural layer, separable from agent logic and from information access, enabling architectural reasoning rather than only engineering productivity. We instantiate this with an information-controlled design on prediction markets: a single LLM, fixed tools, fixed per-call output cap, and fixed prompt template across five reference coordination configurations, with total compute per question treated as an endogenous architectural output. The Murphy decomposition of the Brier score separates calibration from discriminative power, so configurations leave distinguishable signatures even when aggregate scores coincide. On 100 Polymarket binary markets resolved after the model’s training cutoff (claude-opus-4-6) we report Murphy signatures, a cost-quality Pareto frontier, category-conditioned analysis, and a bootstrap power-projection. Three of five pre-specified predictions are upheld in direction; two configurations dominate the Pareto frontier within this regime; exploratory bootstrap intervals separate consensus alignment from others, though pairwise tests do not survive Bonferroni correction at n=100. We also deploy the same configurations as live agents on Foresight Arena under web-search-enabled conditions, as an on-chain replication channel accumulating in parallel. Harness, trace dataset, and production agents are released. We position this as a methodology-validating first instantiation, not a general cross-model claim.
Complexity vs Empirical Score
- Math Complexity: 5.5/10
- Empirical Rigor: 7.5/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper combines formal, rigorous statistical methods like the Murphy decomposition and power projection analysis with a structured empirical approach, including live on-chain agent deployment and a controlled experimental design.
Research Flowchart
flowchart TD
A[<b>Goal:</b> Establish Coordination as an Architectural Layer] --> B[<b>Methodology:</b> Information-Controlled Design via 5 Configurations]
C[<b>Inputs:</b> Polymarket Data & Fixed LLM/Tools/Prompts] --> B
B --> D[<b>Process:</b> Murphy Decomposition of Brier Scores]
D --> E[<b>Analysis:</b> Cost-Quality Pareto Frontier & Bootstrap Projections]
E --> F[<b>Outcomes:</b> Coordination Signatures & Methodology Validation]
F --> G[<b>Replication:</b> Live Deployment on Foresight Arena]
Cited References (39)
- Why Do Multi-Agent LLM Systems Fail? (2025) — link
- Semantic Consensus: Process-Aware Conflict Detection and Resolution for Enterprise Multi-Agent LLM Systems (2026) — link
- Talk Isn’t Always Cheap: Understanding Failure Modes in Multi-Agent Debate (2025)
- Rethinking the Value of Multi-Agent Workflow: A Strong Single-Agent Baseline (2026)
- Agentless: Demystifying LLM -Based Software Engineering Agents (2024)
- On the Reliability Limits of LLM -Based Multi-Agent Planning (2026) — link
- A Declarative Language for Building and Orchestrating LLM -Powered Agent Workflows (2025)
- The Auton Agentic AI Framework: A Declarative Architecture for Specification, Governance, and Runtime Execution of Autonomous Agent Systems (2026) — link
and 31 more
- Multi-Agent Collaboration Patterns with Strands Agents and Amazon Nova (2025) — link
- Foundry Agent Service: Workflow Agents and Declarative Orchestration (2026) — link
- HuggingGPT : Solving AI Tasks with ChatGPT and Its Friends in Hugging Face (2023)
- AutoGen : Enabling Next-Gen LLM Applications via Multi-Agent Conversation (2023)
- LLM -Coordination: Evaluating and Analyzing Multi-Agent Coordination Abilities in Large Language Models (2025)
- An Introduction to MultiAgent Systems (2009)
- Multiagent Systems: A Survey from a Machine Learning Perspective (2000)
- Design Patterns: Elements of Reusable Object-Oriented Software (1994)
- Generative Communication in Linda (1985)
- Communicating and Mobile Systems: The $\pi$ -calculus (1999)
- Communicating Sequential Processes (1985)
- A Universal Modular ACTOR Formalism for Artificial Intelligence (1973)
- Verification of Forecasts Expressed in Terms of Probability (1950)
- A New Vector Partition of the Probability Score (1973)
- Strictly Proper Scoring Rules, Prediction, and Estimation (2007)
- The Comparison and Evaluation of Forecasters (1983)
- Prediction Markets (2004)
- Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggregation (2007)
- Superforecasting: The Art and Science of Prediction (2015)
- Approaching Human-Level Forecasting with Language Models (2024)
- Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy (2024)
- Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournament (2023)
- ForecastBench : A Dynamic Benchmark of AI Forecasting Capabilities (2024)
- Prediction Arena : Benchmarking AI Models on Real-World Prediction Markets (2026)
- SWE-bench : Can Language Models Resolve Real-World GitHub Issues? (2024)
- Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents (2026)
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2023)
- Model Context Protocol Specification (2024) — link
- Statistical Power Analysis for the Behavioral Sciences (1988)
- Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors (2014)
- Datasheets for Datasets (2021)
0 of 39 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.