Paper: arXiv 2501.12399

Authors: Shijie Han, Jingshu Zhang, Yiqing Shen, Kaiyuan Yan, Hongguang Li

Abstract

Current financial large language models (FinLLMs) struggle with two critical limitations: the absence of objective evaluation metrics to assess the quality of stock analysis reports and a lack of depth in stock analysis, which impedes their ability to generate professional-grade insights. To address these challenges, this paper introduces FinSphere, a stock analysis agent, along with three major contributions: (1) AnalyScore, a systematic evaluation framework for assessing stock analysis quality, (2) Stocksis, a dataset curated by industry experts to enhance LLMs’ stock analysis capabilities, and (3) FinSphere, an AI agent that can generate high-quality stock analysis reports in response to user queries. Experiments demonstrate that FinSphere achieves superior performance compared to both general and domain-specific LLMs, as well as existing agent-based systems, even when they are enhanced with real-time data access and few-shot guidance. The integrated framework, which combines real-time data feeds, quantitative tools, and an instruction-tuned LLM, yields substantial improvements in both analytical quality and practical applicability for real-world stock analysis.

Complexity vs Empirical Score

  • Math Complexity: 2.5/10
  • Empirical Rigor: 9.0/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper’s mathematics is minimal, focusing on evaluation metrics and structured frameworks rather than complex formulas or derivations. In contrast, it demonstrates high empirical rigor by introducing a new dataset (Stocksis), detailed evaluation benchmarks (AnalyScore), real-time data integration, and quantitative tool usage, all backed by comparative experiments against general and domain-specific LLMs.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Address FinLLM limitations:<br>evaluation & depth"] --> B["Methodology & Components"]
  
  B --> C{"Dataset & Evaluation"}
  B --> D{"AI Agent System"}
  
  C --> E["Stocksis Dataset<br>Curated by experts"]
  C --> F["AnalyScore Framework<br>Systematic evaluation metrics"]
  
  D --> G["FinSphere Agent<br>Integration of LLM, tools, & data"]
  D --> H["Real-time Data Feeds<br>& Quantitative Tools"]
  
  E --> G
  F --> G
  H --> G
  
  G --> I["Key Findings & Outcomes"]
  
  subgraph I ["Superior Performance"]
      J["Outperforms general LLMs"]
      K["Outperforms domain-specific LLMs"]
      L["Beats agent-based systems<br>even with real-time data & few-shot"]
      M["Higher analytical quality<br>& practical applicability"]
  end
  
  J --> I
  K --> I
  L --> I
  M --> I

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes FinSphere, an instruction‑tuned real‑time stock analysis agent, together with the AnalyScore evaluation framework and the Stocksis dataset for enhancing LLM stock analysis.

Source sentence
this paper introduces FinSphere, a stock analysis agent, along with three major contributions: (1) AnalyScore, a systematic evaluation framework for assessing stock analysis quality, (2) Stocksis, a dataset curated by industry experts to enhance LLMs’ stock analysis capabilities, and (3) FinSphere, an AI agent that can

Assumes 2

  • The system assumes access to continuously updated real‑time financial databases and quantitative tools. data
    Source sentence
    real-time databases and quantitative tools for dynamic and actionable stock analysis.
  • Full‑parameter instruction tuning can internalize complex analytical patterns for professional reporting. model
    Source sentence
    instruction tuning enables the model to internalize complex analytical patterns and generate coherent, multi‑dimensional reports.

Data

  • Stocksis text-news unknown
    Source sentence
    Stocksis, a specialized dataset curated by industry experts to enhance LLMs’ financial question‑answering and stock analysis capabilities.

Costs

Not applicable The paper reports no trading or execution result.

Finds 4

  • FinSphere achieves superior performance compared to both general and domain‑specific LLMs and existing agent‑based systems. performance
    Source sentence
    superior performance in generating comprehensive stock analyses.
  • Average Kendall’s Tau across 100 queries for different annotator groups ranges from 71.45% to 94.25%, indicating strong agreement in evaluation. statistical
    Source sentence
    Results show strong agreement, with most correlations exceeding 80% and values ranging from 71.45 to 94.25.
  • Ablation study shows overall scores increase from 58.90 (20% of Stocksis) to 70.88 (100% of Stocksis), demonstrating a positive correlation between training data scale and performance. performance
    Source sentence
    overall scores increasing from 58.90 (20%) to 70.88 (100%).
  • FinSphere’s Expression dimension score (14.87) is noticeably higher than FinMem (12.85) and FinRobot (11.55), highlighting superior logical articulation. performance
    Source sentence
    FinSphere achieves 14.87, noticeably higher than FinMem (12.85) and FinRobot (11.55).

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 25 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (49)

  1. Large language models in finance: A survey — → 2311.10723
  2. Data-centric FinGPT: Democratizing Internet-scale Data for Financial Large Language Models (2023) — link
  3. PIXIU: a large language model, instruction data and evaluation benchmark for finance (2023)
  4. TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance (2021)
  5. Instruct-fingpt: Financial sentiment analysis by instruction tuning of general-purpose large language models — → 2306.12659
  6. FinQA: A Dataset of Numerical Reasoning over Financial Data (2021)
  7. Qwen2 technical report (2024) — link
  8. Gpt-4 technical report (2023) — link
and 41 more
  1. FinGPT: Open-Source Financial Large Language Models — → 2306.06031
  2. LLM-empowered Chatbots for More Accessible Financial Services (2023) — link
  3. Integrating Large Language Models with Graphical Models for Decision Making in Finance (2023) — link
  4. FinOps: An Open-source Framework for LLM-based Financial Agent (2023) — link
  5. A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist — → 2402.18485
  6. Portfolio performance based on LLM news scores and related economical analysis (2024)
  7. TradingGPT: Multi-agent system with layered memory and distinct characters for enhanced financial trading performance — → 2309.03736
  8. Learning to generate explainable stock predictions using self-reflective large language models — → 2402.03659
  9. Revolutionizing finance with llms: An overview of applications and insights (2024) — link
  10. Enhancing Anomaly Detection in Financial Markets with an LLM-based Multi-Agent Framework — → 2403.19735
  11. GPT-InvestAR: Enhancing stock investment strategies through annual report analysis with large language models — → 2309.03079
  12. Large Language Model Agent in Financial Trading: A Survey — → 2408.06361
  13. Stock Price Trend Prediction using Emotion Analysis of Financial Headlines with Distilled LLM Model (2024)
  14. DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services (2023)
  15. The finben: An holistic financial benchmark for large language models (2024) — link
  16. CFBenchmark: Chinese financial assistant benchmark for large language model (2023) — link
  17. FinRL-meta: A universe of near-real market environments for data-driven deep reinforcement learning in quantitative finance (2021) — link
  18. Harnessing earnings reports for stock predictions: A qlora-enhanced llm approach — → 2408.06634
  19. Bloomberggpt: A large language model for finance (2023) — link
  20. FinMem: A performance-enhanced LLM trading agent with layered memory and character design — → 2311.13743
  21. Fineval: A chinese financial domain knowledge evaluation benchmark for large language models (2023) — link
  22. Financebench: A new benchmark for financial question answering (2023) — link
  23. FinTextQA: A Dataset for Long-form Financial Question Answering (2024) — link
  24. Deepseek-v3 technical report (2024) — link
  25. LoRA: Low-Rank Adaptation of Large Language Models (2021)
  26. FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models — → 2405.14767
  27. The llama 3 herd of models (2024) — link
  28. XBRL Agent: Leveraging Large Language Models for Financial Report Analysis (2024)
  29. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
  30. A package for automatic evaluation of summaries (2004)
  31. Bleu: a method for automatic evaluation of machine translation (2002)
  32. Large language models and generative AI in finance: an analysis of ChatGPT, Bard, and Bing AI (2023)
  33. Large Language Models and Sentiment Analysis in Financial Markets: A Review, Datasets and Case Study (2024)
  34. Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications — → 2408.11878
  35. Investlm: A large language model for investment using financial domain instruction tuning — → 2309.13064
  36. Finbert: A pre-trained financial language representation model for financial text mining (2021)
  37. A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges — → 2406.11903
  38. Finbert: A pretrained language model for financial communications (2020) — link
  39. Financial statement analysis with large language models — → 2407.17866
  40. FinBERT: A large language model for extracting information from financial text (2023)
  41. Extracting financial data from unstructured sources: Leveraging large language models (2024)

16 of 49 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.