Paper: arXiv 2603.19225

Authors: Yogesh Agrawal, Aniruddha Dutta, Md Mahadi Hasan, Santu Karmaker, Aritra Dutta

Abstract

Real-world financial decision-making is a challenging problem that requires reasoning over heterogeneous signals, including company fundamentals derived from regulatory filings and trading signals computed from price dynamics. Recently, with the advancement of Large Language Models (LLMs), financial analysts have begun to use them for financial decision-making tasks. However, existing financial question answering benchmarks for testing these models primarily focus on company balance sheet data and rarely evaluate reasoning over how company stocks trade in the market or their interactions with fundamentals. To take advantage of the strengths of both approaches, we introduce FinTradeBench, a benchmark for evaluating financial reasoning that integrates company fundamentals and trading signals. FinTradeBench contains 1,400 questions grounded in NASDAQ-100 companies over a ten-year historical window. The benchmark is organized into three reasoning categories: fundamentals-focused, trading-signal-focused, and hybrid questions requiring cross-signal reasoning. To ensure reliability at scale, we adopt a calibration-then-scaling framework that combines expert seed questions, multi-model response generation, intra-model self-filtering, numerical auditing, and human-LLM judge alignment. We evaluate 14 LLMs under zero-shot prompting and retrieval-augmented settings and witness a clear performance gap. Retrieval substantially improves reasoning over textual fundamentals, but provides limited benefit for trading-signal reasoning. These findings highlight fundamental challenges in the numerical and time-series reasoning for current LLMs and motivate future research in financial intelligence.

Complexity vs Empirical Score

  • Math Complexity: 3.0/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper introduces an empirical evaluation benchmark (FinTradeBench) for LLMs using 1,400 multi-modal financial questions across NASDAQ-100 companies. While heavy on empirical methodology, dataset scaling, and model benchmarking across 14 LLMs, it features low mathematical complexity.

Research Flowchart

  flowchart TD
    A[Goal: Evaluate LLM Reasoning over Financial Fundamentals & Trading Signals] --> B[Dataset Construction: 1,400 Questions on NASDAQ-100 Companies over 10 Years]
    B --> C[Framework: Calibration-then-Scaling <br><i>Seeds, Self-Filtering, Auditing, Human-LLM Alignment</i>]
    C --> D[Categorization: Fundamentals, Trading Signals, and Hybrid Questions]
    D --> E[Evaluation: Test 14 LLMs under Zero-Shot and RAG Settings]
    E --> F[Findings: RAG helps textual fundamentals but has limited benefit for trading signals]
    E --> G[Findings: Highlighted gaps in numerical and time-series financial reasoning]