Paper: arXiv 2409.07487

Authors: Sandy Chen, Leqi Zeng, Abhinav Raghunathan, Flora Huang, Terrence C. Kim

Abstract

Large Language Models (LLMs) research in the financial domain is particularly complex due to the sheer number of approaches proposed in literature. Retrieval-Augmented Generation (RAG) has emerged as one of the leading methods in the sector due to its inherent groundedness and data source variability. In this work, we introduce a RAG framework called Mixture of Agents (MoA) and demonstrate its viability as a practical, customizable, and highly effective approach for scaling RAG applications. MoA is essentially a layered network of individually customized small language models (Hoffmann et al., 2022) collaborating to answer questions and extract information. While there are many theoretical propositions for such an architecture and even a few libraries for generally applying the structure in practice, there are limited documented studies evaluating the potential of this framework considering real business constraints such as cost and speed. We find that the MoA framework, consisting of small language models (Hoffmann et al., 2022), produces higher quality and more grounded responses across various financial domains that are core to Vanguard’s business while simultaneously maintaining low costs.

Complexity vs Empirical Score

  • Math Complexity: 2.0/10
  • Empirical Rigor: 3.0/10
  • Quadrant: Philosophers — conceptual discussion, lower math and empirical scores

Why this score: The paper focuses on system architecture (MoA) and RAG frameworks with minimal mathematical derivations or formulas, relying instead on conceptual comparisons and enterprise application descriptions. While it claims cost and quality improvements, the excerpt provides no backtested datasets, statistical metrics, or implementation code, relying on high-level assertions rather than rigorous empirical validation.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Develop Cost-Effective RAG<br>for Financial NLP"] --> B["Methodology:<br>Mixture of Agents (MoA)<br>Layered Architecture"]
  
  B --> C["Input Data:<br>Financial Documents &<br>Vanguard Business Queries"]
  
  C --> D["Process:<br>Collaborative Small LLMs<br>Chained Retrieval & Generation"]
  
  D --> E["Key Constraints:<br>Cost & Speed Evaluation"]
  
  E --> F["Outcomes:<br>Higher Quality Grounded Responses<br>Low Operational Cost"]
  
  F --> G["Outcome:<br>Viable Framework for<br>Scaling RAG Applications"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper introduces a Retrieval-Augmented Generation framework called Mixture of Agents (MoA), a layered network of customized small language models that collaborate to answer questions and extract information.

Source sentence
In this work, we introduce a RAG framework called Mixture of Agents (MoA) and demonstrate its viability as a practical, customizable, and highly effective approach for scaling RAG applications.

Assumes 2

  • The evaluation assumes that cost and speed are primary constraints for enterprise RAG deployments. execution
    Source sentence
    While there are many theoretical propositions for such an architecture and even a few libraries for generally applying the structure in practice, there are limited documented studies evaluating the potential of this framework considering real business constraints such as cost and speed. Although libraries such as AIFlows, Langchain, and Microsoft Autogen enable programmatic composition of agents and LLMs, there are
  • Small language models (sub‑10B parameters) can provide high quality, grounded responses at lower cost than larger models. model
    Source sentence
    We find that the MoA framework, consisting of small language models (Hoffmann et al., 2022), produces higher quality and more grounded responses across various financial domains that are core to Vanguard's business while simultaneously maintaining low costs. At Vanguard, we have supported the hypothesis that smaller language models [1] are the present and future when it comes to highly efficient and accurate

Data

  • 10‑K/Q filings · Vanguard internal document repository text-news proprietary
    Source sentence
    Additionally, it has RAG access to raw filings and API access to a SQL database containing analyst notes with domain-specific equations.
  • Analyst notes with domain‑specific equations · Vanguard internal database text-news proprietary
    Source sentence
    Additionally, it has RAG access to raw filings and API access to a SQL database containing analyst notes with domain-specific equations.

Costs

Not applicable The paper reports no trading or execution result.

Finds 5

  • MoA produces higher quality and more grounded responses across various financial domains while maintaining low costs. performance
    Source sentence
    We find that the MoA framework, consisting of small language models (Hoffmann et al., 2022), produces higher quality and more grounded responses across various financial domains that are core to Vanguard's business while simultaneously maintaining low costs.
  • The latency penalty for MoA is approximately 4.07× (or 2.24× with parallel inference) compared with the original single‑model system. performance
    Source sentence
    The latency penalty for implementing MoA is approximately 4.07x, or 2.24x when running inference in parallel.In comparison, our original single-model system was capable of performing the 8 same operation in under three seconds.
  • MoA has scaled to analyzing tens of thousands of documents simultaneously. performance
    Source sentence
    At Vanguards IMFS team, our MoA system has scaled to analyzing tens of thousands of documents simultaneously.
  • Compounding error occurs with single‑stream serial models but not with MoA. theoretical
    Source sentence
    In-terestingly, we observe that the concept of ”compounding error” only occurs with a single stream of serial models and not with MoA.
  • MoA can operate at a fraction of the cost by utilizing open‑weight, sub‑10B parameter models. performance
    Source sentence
    MoA is an extension of this hypothesis, as it has allowed us to operate at a fraction of the cost by utilizing open-weight, sub-10B parameter models.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 33 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (17)

  1. Flows: Building Blocks of Reasoning and Collaborating AI (2024) — link
  2. Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024) — link
  3. Simulating Opinion Dynamics with Networks of LLM-based Agents (2023) — link
  4. Lost in the Middle: How Language Models Use Long Contexts (2023) — link
  5. More Agents Is All You Need (2024) — link
  6. Multi-AI collaboration helps reasoning and factual accuracy in large language models (2023) — link
  7. Learning to Decode Collaboratively with Multiple Language Models (2024) — link
  8. Mixtral of Experts (2024) — link
and 9 more
  1. SocraticAI — link
  2. Unlocking the Power of Multiple Language Models: A Dive into Collaborative AI (2023) — link
  3. EntGPT: Linking Generative Large Language Models with Knowledge Bases (2024) — link
  4. Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023) — link
  5. Retrieval-Augmented Generation for Large Language Models: A Survey (2023) — link
  6. Introduction | ��️�� Langchain — link
  7. Training Compute-Optimal Large Language Models (2022) — link
  8. Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022) — link
  9. Mixture-of-Agents Enhances Large Language Model Capabilities (2024) — link

0 of 17 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.