Paper: arXiv 2501.12399
Authors: Shijie Han, Jingshu Zhang, Yiqing Shen, Kaiyuan Yan, Hongguang Li
Abstract
Current financial large language models (FinLLMs) struggle with two critical limitations: the absence of objective evaluation metrics to assess the quality of stock analysis reports and a lack of depth in stock analysis, which impedes their ability to generate professional-grade insights. To address these challenges, this paper introduces FinSphere, a stock analysis agent, along with three major contributions: (1) AnalyScore, a systematic evaluation framework for assessing stock analysis quality, (2) Stocksis, a dataset curated by industry experts to enhance LLMs’ stock analysis capabilities, and (3) FinSphere, an AI agent that can generate high-quality stock analysis reports in response to user queries. Experiments demonstrate that FinSphere achieves superior performance compared to both general and domain-specific LLMs, as well as existing agent-based systems, even when they are enhanced with real-time data access and few-shot guidance. The integrated framework, which combines real-time data feeds, quantitative tools, and an instruction-tuned LLM, yields substantial improvements in both analytical quality and practical applicability for real-world stock analysis.
Complexity vs Empirical Score
- Math Complexity: 2.5/10
- Empirical Rigor: 9.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper’s mathematics is minimal, focusing on evaluation metrics and structured frameworks rather than complex formulas or derivations. In contrast, it demonstrates high empirical rigor by introducing a new dataset (Stocksis), detailed evaluation benchmarks (AnalyScore), real-time data integration, and quantitative tool usage, all backed by comparative experiments against general and domain-specific LLMs.
Research Flowchart
flowchart TD
A["Research Goal<br>Address FinLLM limitations:<br>evaluation & depth"] --> B["Methodology & Components"]
B --> C{"Dataset & Evaluation"}
B --> D{"AI Agent System"}
C --> E["Stocksis Dataset<br>Curated by experts"]
C --> F["AnalyScore Framework<br>Systematic evaluation metrics"]
D --> G["FinSphere Agent<br>Integration of LLM, tools, & data"]
D --> H["Real-time Data Feeds<br>& Quantitative Tools"]
E --> G
F --> G
H --> G
G --> I["Key Findings & Outcomes"]
subgraph I ["Superior Performance"]
J["Outperforms general LLMs"]
K["Outperforms domain-specific LLMs"]
L["Beats agent-based systems<br>even with real-time data & few-shot"]
M["Higher analytical quality<br>& practical applicability"]
end
J --> I
K --> I
L --> I
M --> I
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes FinSphere, an instruction‑tuned real‑time stock analysis agent, together with the AnalyScore evaluation framework and the Stocksis dataset for enhancing LLM stock analysis.
Source sentence
this paper introduces FinSphere, a stock analysis agent, along with three major contributions: (1) AnalyScore, a systematic evaluation framework for assessing stock analysis quality, (2) Stocksis, a dataset curated by industry experts to enhance LLMs’ stock analysis capabilities, and (3) FinSphere, an AI agent that can
Assumes 2
- The system assumes access to continuously updated real‑time financial databases and quantitative tools. data
Source sentence
real-time databases and quantitative tools for dynamic and actionable stock analysis.
- Full‑parameter instruction tuning can internalize complex analytical patterns for professional reporting. model
Source sentence
instruction tuning enables the model to internalize complex analytical patterns and generate coherent, multi‑dimensional reports.
Data
- Stocksis text-news unknown
Source sentence
Stocksis, a specialized dataset curated by industry experts to enhance LLMs’ financial question‑answering and stock analysis capabilities.
Costs
Not applicable
Finds 4
- FinSphere achieves superior performance compared to both general and domain‑specific LLMs and existing agent‑based systems. performance
Source sentence
superior performance in generating comprehensive stock analyses.
- Average Kendall’s Tau across 100 queries for different annotator groups ranges from 71.45% to 94.25%, indicating strong agreement in evaluation. statistical
Source sentence
Results show strong agreement, with most correlations exceeding 80% and values ranging from 71.45 to 94.25.
- Ablation study shows overall scores increase from 58.90 (20% of Stocksis) to 70.88 (100% of Stocksis), demonstrating a positive correlation between training data scale and performance. performance
Source sentence
overall scores increasing from 58.90 (20%) to 70.88 (100%).
- FinSphere’s Expression dimension score (14.87) is noticeably higher than FinMem (12.85) and FinRobot (11.55), highlighting superior logical articulation. performance
Source sentence
FinSphere achieves 14.87, noticeably higher than FinMem (12.85) and FinRobot (11.55).
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 25 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (49)
- Large language models in finance: A survey — → 2311.10723
- Data-centric FinGPT: Democratizing Internet-scale Data for Financial Large Language Models (2023) — link
- PIXIU: a large language model, instruction data and evaluation benchmark for finance (2023)
- TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance (2021)
- Instruct-fingpt: Financial sentiment analysis by instruction tuning of general-purpose large language models — → 2306.12659
- FinQA: A Dataset of Numerical Reasoning over Financial Data (2021)
- Qwen2 technical report (2024) — link
- Gpt-4 technical report (2023) — link
and 41 more
- FinGPT: Open-Source Financial Large Language Models — → 2306.06031
- LLM-empowered Chatbots for More Accessible Financial Services (2023) — link
- Integrating Large Language Models with Graphical Models for Decision Making in Finance (2023) — link
- FinOps: An Open-source Framework for LLM-based Financial Agent (2023) — link
- A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist — → 2402.18485
- Portfolio performance based on LLM news scores and related economical analysis (2024)
- TradingGPT: Multi-agent system with layered memory and distinct characters for enhanced financial trading performance — → 2309.03736
- Learning to generate explainable stock predictions using self-reflective large language models — → 2402.03659
- Revolutionizing finance with llms: An overview of applications and insights (2024) — link
- Enhancing Anomaly Detection in Financial Markets with an LLM-based Multi-Agent Framework — → 2403.19735
- GPT-InvestAR: Enhancing stock investment strategies through annual report analysis with large language models — → 2309.03079
- Large Language Model Agent in Financial Trading: A Survey — → 2408.06361
- Stock Price Trend Prediction using Emotion Analysis of Financial Headlines with Distilled LLM Model (2024)
- DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services (2023)
- The finben: An holistic financial benchmark for large language models (2024) — link
- CFBenchmark: Chinese financial assistant benchmark for large language model (2023) — link
- FinRL-meta: A universe of near-real market environments for data-driven deep reinforcement learning in quantitative finance (2021) — link
- Harnessing earnings reports for stock predictions: A qlora-enhanced llm approach — → 2408.06634
- Bloomberggpt: A large language model for finance (2023) — link
- FinMem: A performance-enhanced LLM trading agent with layered memory and character design — → 2311.13743
- Fineval: A chinese financial domain knowledge evaluation benchmark for large language models (2023) — link
- Financebench: A new benchmark for financial question answering (2023) — link
- FinTextQA: A Dataset for Long-form Financial Question Answering (2024) — link
- Deepseek-v3 technical report (2024) — link
- LoRA: Low-Rank Adaptation of Large Language Models (2021)
- FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models — → 2405.14767
- The llama 3 herd of models (2024) — link
- XBRL Agent: Leveraging Large Language Models for Financial Report Analysis (2024)
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- A package for automatic evaluation of summaries (2004)
- Bleu: a method for automatic evaluation of machine translation (2002)
- Large language models and generative AI in finance: an analysis of ChatGPT, Bard, and Bing AI (2023)
- Large Language Models and Sentiment Analysis in Financial Markets: A Review, Datasets and Case Study (2024)
- Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications — → 2408.11878
- Investlm: A large language model for investment using financial domain instruction tuning — → 2309.13064
- Finbert: A pre-trained financial language representation model for financial text mining (2021)
- A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges — → 2406.11903
- Finbert: A pretrained language model for financial communications (2020) — link
- Financial statement analysis with large language models — → 2407.17866
- FinBERT: A large language model for extracting information from financial text (2023)
- Extracting financial data from unstructured sources: Leveraging large language models (2024)
16 of 49 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.