Paper: arXiv 2507.22936

Authors: Md Talha Mohsin

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide variety of Financial Natural Language Processing (FinNLP) tasks. However, systematic comparisons among widely used LLMs remain underexplored. Given the rapid advancement and growing influence of LLMs in financial analysis, this study conducts a thorough comparative evaluation of five leading LLMs, GPT, Claude, Perplexity, Gemini and DeepSeek, using 10-K filings from the ‘Magnificent Seven’ technology companies. We create a set of domain-specific prompts and then use three methodologies to evaluate model performance: human annotation, automated lexical-semantic metrics (ROUGE, Cosine Similarity, Jaccard), and model behavior diagnostics (prompt-level variance and across-model similarity). The results show that GPT gives the most coherent, semantically aligned, and contextually relevant answers; followed by Claude and Perplexity. Gemini and DeepSeek, on the other hand, have more variability and less agreement. Also, the similarity and stability of outputs change from company to company and over time, showing that they are sensitive to how prompts are written and what source material is used.

Complexity vs Empirical Score

  • Math Complexity: 2.5/10
  • Empirical Rigor: 6.0/10 (capped from 8.0: no out-of-sample test; see penalty flags)
  • Quadrant: Street Traders — empirical and applied, lower math complexity (effective empirical rigor capped from 8.0)

Why this score: The paper’s mathematics is minimal, relying on standard statistical metrics like ROUGE and Cosine Similarity, but its empirical rigor is high, involving a systematic comparison of five LLMs on real financial data (10-K filings), domain-specific prompting, and multiple evaluation methodologies.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Evaluate LLMs on Financial NLP<br>for 10-K Report Analysis"] --> B["Data Source:<br>10-K Filings from<br>'Magnificent Seven' Tech Firms"]
  B --> C["Methodology:<br>3-Pronged Evaluation"]
  C --> D["Human Annotation<br>(Qualitative Assessment)"]
  C --> E["Automated Metrics<br>ROUGE / Cosine / Jaccard"]
  C --> F["Diagnostics<br>Variance & Similarity"]
  D & E & F --> G["Key Findings"]
  G --> H1["GPT: Best Coherence & Alignment"]
  G --> H2["Claude/Perplexity: Moderate Performance"]
  G --> H3["Gemini/DeepSeek: High Variability"]
  G --> H4["Performance Sensitive to Prompting & Context"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a controlled evaluation of five transformer‑based LLMs on question answering over 10‑K Business sections, using human evaluation, automated similarity metrics, and behavioral diagnostics.

Source sentence
This paper presents a controlled evaluation of five transformer-based LLMs applied to question answering over the Business sections of U.S. 10‑K filings.

Assumes 2

  • The Business sections of 10‑K filings are representative of complex financial disclosures for evaluating LLM behavior. data
    Source sentence
    Focusing on large, information‑rich disclosures from major technology firms allows for a controlled and demanding evaluation setting
  • Chain‑of‑Thought prompting can make LLMs behave like financial analysts, enabling simulation of real‑life analytical workflows. model
    Source sentence
    We introduce a Chain‑of‑Thought (CoT) prompting approach that tells models to behave like financial analysts in order to simulate real‑life analytical workflows.

Data

  • Business section of 10‑K filings · U.S. Securities and Exchange Commission (SEC) text-news public
    last three years · "Magnificent 7" tech companies
    Source sentence
    ChatGPT‑4, Perplexity, Claude 4 Opus, Gemini, and DeepSeek—look at the Business section (Item 1) of 10‑K filings from the ”Magnificent 7” tech companies over the last three years.

Recognised sources: SEC EDGAR

Costs

Not applicable The paper reports no trading or execution result.

Finds 3

  • No single LLM consistently dominates across all evaluation perspectives; performance differences are relative tendencies under the tested conditions. other
    Source sentence
    Importantly, no single model consistently dominates across all evaluation perspectives.
  • GPT achieved the highest average human scores on relevance, completeness, clarity, and factual accuracy among the models evaluated. performance
    Source sentence
    GPT is rated by the highest number of annotators on the scale of relevance, completeness, and clarity. It also scores well on the accuracy of the facts.
  • Inter‑rater reliability was low across all qualitative dimensions, indicating substantial disagreement among expert annotators. statistical
    Source sentence
    Agreement levels are generally low across all qualitative dimensions, with α values clustered near zero and occasional negative estimates.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 16 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.