Paper: arXiv 2502.15865

Authors: Zichen Chen, Jiaao Chen, Jianda Chen, Misha Sra

Abstract

Standard benchmarks fixate on how well large language model (LLM) agents perform in finance, yet say little about whether they are safe to deploy. We argue that accuracy metrics and return-based scores provide an illusion of reliability, overlooking vulnerabilities such as hallucinated facts, stale data, and adversarial prompt manipulation. We take a firm position: financial LLM agents should be evaluated first and foremost on their risk profile, not on their point-estimate performance. Drawing on risk-engineering principles, we outline a three-level agenda: model, workflow, and system, for stress-testing LLM agents under realistic failure modes. To illustrate why this shift is urgent, we audit six API-based and open-weights LLM agents on three high-impact tasks and uncover hidden weaknesses that conventional benchmarks miss. We conclude with actionable recommendations for researchers, practitioners, and regulators: audit risk-aware metrics in future studies, publish stress scenarios alongside datasets, and treat safety budget'' as a primary success criterion. Only by redefining what good’’ looks like can the community responsibly advance AI-driven finance.

Complexity vs Empirical Score

  • Math Complexity: 2.0/10
  • Empirical Rigor: 6.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper provides a timely paradigm shift in financial LLM evaluation by prioritizing risk-aware auditing over simple accuracy metrics. While it contains virtually no mathematical derivations, it offers strong conceptual framework completeness, detailed taxonomy of risk, and empirical auditing across six LLM agents.

Research Flowchart

  flowchart TD
    A["<b>Research Goal:</b> Shift Financial LLM Evaluation<br/>from Performance to Risk Profile"] --> B["<b>Core Framework:</b> 3-Level Risk-Engineering Agenda<br/><i>(Model, Workflow, System)</i>"]
    
    B --> C["<b>Audit Inputs & Setup:</b> 6 LLM Agents<br/><i>(API-based & Open-weights)</i><br/>tested on 3 High-Impact Financial Tasks"]
    
    C --> D["<b>Stress-Testing Processes:</b><br/>Simulate Realistic Failures<br/><i>(Hallucinations, Stale Data, Adversarial Prompts)</i>"]
    
    D --> E["<b>Key Findings:</b> Conventional Benchmarks Miss<br/>Critical Vulnerabilities & Hidden Weaknesses"]
    
    E --> F["<b>Actionable Outcomes:</b> Mandate Safety Budgets,<br/>Risk Metrics, & Stress Scenarios for Regulation"]

Cited References (137)

  1. Crafting Papers on Machine Learning (2000)
  2. The Need for Biases in Learning Generalizations (1980)
  3. Computational Complexity of Machine Learning (1989)
  4. Machine Learning: An Artificial Intelligence Approach, Vol. I (1983)
  5. Pattern Classification (2000)
  6. Suppressed for Anonymity (2021)
  7. Mechanisms of Skill Acquisition and the Law of Practice (1981)
  8. Some Studies in Machine Learning Using the Game of Checkers (1959)
and 129 more
  1. FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making (2024) — link
  2. INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent (2024) — link
  3. Simulating Financial Market via Large Language Model based Agents (2024) — link
  4. Enhancing Financial Question Answering with a Multi-Agent Reflection Framework (2024) — link
  5. FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models (2024) — link
  6. Large Language Model Agent in Financial Trading: A Survey (2024) — link
  7. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception (2024) — link
  8. On the effects of data scale on ui control agents (2024)
  9. Screenagent: A vision language model-driven computer control agent (2024) — link
  10. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web (2025)
  11. Mind2web: Towards a generalist agent for the web (2024)
  12. Webshop: Towards scalable real-world web interaction with grounded language agents (2022)
  13. GUICourse: From General Vision Language Models to Versatile GUI Agents (2024) — link
  14. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments (2024) — link
  15. Os-atlas: A foundation action model for generalist gui agents (2024) — link
  16. Seeclick: Harnessing gui grounding for advanced visual gui agents (2024) — link
  17. Mmtom-qa: Multimodal theory of mind question answering (2024) — link
  18. Modelscope-agent: Building your customizable agent system with open-source large language models (2023) — link
  19. Multi-Modal and Multi-Agent Systems Meet Rationality: A Survey (2024) — link
  20. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration (2023) — link
  21. Autogen: Enabling next-gen llm applications via multi-agent conversation framework (2023) — link
  22. Vision-language models can self-improve reasoning via reflection (2024) — link
  23. Language models, agent models, and world models: The law for machine reasoning and planning (2023) — link
  24. A survey of neural code intelligence: Paradigms, advances and beyond (2024) — link
  25. Sciagent: Tool-augmented language models for scientific reasoning (2024) — link
  26. React: Synergizing reasoning and acting in language models (2022) — link
  27. Os-copilot: Towards generalist computer agents with self-improvement (2024) — link
  28. How Far Are We From AGI (2024) — link
  29. OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis (2024) — link
  30. Agent ai: Surveying the horizons of multimodal interaction (2024) — link
  31. A comprehensive overview of large language models (2023) — link
  32. A survey of large language models (2023) — link
  33. Understanding llms: A comprehensive overview from training to inference (2024) — link
  34. A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges — → 2406.11903
  35. Understanding and Mitigating Language Confusion in LLMs (2024)
  36. A framework for measures of risk under uncertainty (2024)
  37. Modern portfolio theory and investment analysis (2009)
  38. The development of utility theory. I (1950)
  39. Quantifying uncertainty in natural language explanations of large language models (2024)
  40. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (2024)
  41. Probabilistic risk analysis: foundations and methods (2001)
  42. Deficiency of large language models in finance: An empirical examination of hallucination (2023)
  43. Hallucination-minimized Data-to-answer Framework for Financial Decision-makers (2023)
  44. Dynamic portfolio optimization across hidden market regimes (2018)
  45. Demystifying Domain-adaptive Post-training for Financial LLMs (2025) — link
  46. Large language models in law: A survey (2024)
  47. Large Language Model in Financial Regulatory Interpretation — → 2405.06808
  48. AI-enhanced blockchain technology: A review of advancements and opportunities (2024)
  49. An LLM can Fool Itself: A Prompt-Based Adversarial Attack (2024)
  50. Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition (2023) — link
  51. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework (2024)
  52. Optimizing LLM based retrieval augmented generation pipelines in the financial domain (2024)
  53. Benchmarking Large Language Model Volatility — → 2311.15180
  54. Risk management in the artificial intelligence act (2024)
  55. Walking a Tightrope–Evaluating Large Language Models in High-Risk Domains (2023)
  56. FinAgent: A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist — → 2402.18485
  57. ADESSE: Advice Explanations in Complex Repeated Decision-Making Environments (2024) — link
  58. ShortcutsBench: A Large-Scale Real-world Benchmark for API -based Agents (2025) — link
  59. Attacks on Third-Party APIs of Large Language Models (2024)
  60. Price discovery and the accuracy of consolidated data feeds in the US equity markets (2018)
  61. Bloomberggpt: A large language model for finance (2023) — link
  62. FinGPT: Democratizing Internet-scale Data for Financial Large Language Models (2023)
  63. Gpt-4 technical report (2023) — link
  64. Fine-Tuning Large Language Models for Stock Return Prediction Using Newsflow — → 2407.18103
  65. Scaling Core Earnings Measurement with Large Language Models (2024)
  66. Financial statement analysis with large language models — → 2407.17866
  67. Pixiu: A comprehensive benchmark, instruction dataset and large language model for finance (2024)
  68. MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model — → 2409.07486
  69. Opportunities and challenges of generative-AI in finance (2024)
  70. Inadequacies of large language model benchmarks in the era of generative artificial intelligence (2024) — link
  71. The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4 (2023) — link
  72. Large Language Models Meet NLP: A Survey (2024) — link
  73. LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks (2024)
  74. Smart contract vulnerability detection: The role of large language model (llm) (2024)
  75. Human player outwits freysa ai agent in $47,000 crypto challenge (2024)
  76. ChatGPT’s Solana API Suggestion Links to Phishing Site, Costing User $2 , 500 (2024)
  77. Unleashing the Potential of Large Language Models for Predictive Tabular Tasks in Data Science (2024) — link
  78. To Believe or Not to Believe Your LLM (2024) — link
  79. Can We Trust LLMs? Mitigate Overconfidence Bias in LLMs through Knowledge Transfer (2024) — link
  80. Are Large Language Models Rational Investors? (2024) — link
  81. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions (2024)
  82. Investigating online financial misinformation and its consequences: A computational perspective (2023) — link
  83. Are large language models temporally grounded? (2023) — link
  84. PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (2024) — link
  85. Revolutionizing finance with llms: An overview of applications and insights (2024) — link
  86. A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law (2024) — link
  87. Basel III: an overview (2011)
  88. Software testing with large language models: Survey, landscape, and vision (2024)
  89. Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022) — link
  90. R -Judge: Benchmarking Safety Risk Awareness for LLM Agents (2024) — link
  91. GLUE : A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2019) — link
  92. TradingAgents: Multi-Agents LLM Financial Trading Framework (2025) — link
  93. The finben: An holistic financial benchmark for large language models (2024) — link
  94. Financebench: A new benchmark for financial question answering (2023) — link
  95. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models (2024)
  96. Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting (2024)
  97. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (2024) — link
  98. A causal explainable guardrails for large language models (2024)
  99. A brief survey on safety of large language models (2024)
  100. Deductive Verification of Chain-of-Thought Reasoning (2023) — link
  101. Training language models to follow instructions with human feedback (2022)
  102. Large language models in finance: A survey (2023)
  103. Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023) — link
  104. Efficient adversarial training in llms with continuous attacks (2024) — link
  105. Engaging with AI: How Interface Design Shapes Human-AI Collaboration in High-Stakes Decision-Making (2025) — link
  106. Yahoo Finance (2024) — link
  107. The llama 3 herd of models (2024) — link
  108. The Claude 3 Model Family: Opus, Sonnet, Haiku — link
  109. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025) — link
  110. D 'o lares or dollars? unraveling the bilingual prowess of financial llms between spanish and english (2024)
  111. Construction of a J apanese Financial Benchmark for Large Language Models (2024) — link
  112. E con L ogic QA : A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning (2024) — link
  113. AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework (2024)
  114. Explainable artificial intelligence (XAI) in finance: a systematic literature review (2024)
  115. Explainable machine learning in credit risk management (2021)
  116. Regulating artificial intelligence in finance: Putting the human in the loop (2021)
  117. KnowledgeMath: Knowledge-Intensive Math Word Problem Solving in Finance Domains (2023) — link
  118. Exploring large language model based intelligent agents: Definitions, methods, and prospects (2024) — link
  119. Is the 2007 US sub-prime financial crisis so different? An international historical comparison (2008)
  120. Model context protocol (mcp): Landscape, security threats, and future research directions (2025) — link
  121. Agent2Agent (A2A) Protocol Documentation (2025)
  122. AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways (2025) — link
  123. Engineering a safer world: Systems thinking applied to safety (2016)
  124. Artificial intelligence risk management framework (AI RMF 1.0) (2023)
  125. Modeling and hazard analysis using STPA (2010)
  126. Qwen3 technical report (2025) — link
  127. Large Language Model Safety: A Holistic Survey (2024) — link
  128. Safety at Scale: A Comprehensive Survey of Large Model Safety (2025) — link
  129. A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment (2025) — link

7 of 137 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.