Paper: arXiv 2410.14059
Authors: Yuzhe Yang, Yifei Zhang, Yan Hu, Yilin Guo, Ruoli Gan, Yueru He, Mingcong Lei, Xiao Zhang, Haining Wang, Qianqian Xie, Jimin Huang, Honghai Yu, Benyou Wang
Abstract
This paper introduces the UCFE: User-Centric Financial Expertise benchmark, an innovative framework designed to evaluate the ability of large language models (LLMs) to handle complex real-world financial tasks. UCFE benchmark adopts a hybrid approach that combines human expert evaluations with dynamic, task-specific interactions to simulate the complexities of evolving financial scenarios. Firstly, we conducted a user study involving 804 participants, collecting their feedback on financial tasks. Secondly, based on this feedback, we created our dataset that encompasses a wide range of user intents and interactions. This dataset serves as the foundation for benchmarking 11 LLMs services using the LLM-as-Judge methodology. Our results show a significant alignment between benchmark scores and human preferences, with a Pearson correlation coefficient of 0.78, confirming the effectiveness of the UCFE dataset and our evaluation approach. UCFE benchmark not only reveals the potential of LLMs in the financial domain but also provides a robust framework for assessing their performance and user satisfaction.
Complexity vs Empirical Score
- Math Complexity: 2.0/10
- Empirical Rigor: 8.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper introduces a new evaluation benchmark with a user study, dataset creation, and a rigorous correlation analysis (Pearson r=0.78), but contains almost no advanced mathematics or formal modeling.
Research Flowchart
flowchart TD A["Research Goal<br>Evaluate LLM Financial Expertise<br>User-Centric Approach"] --> B["Methodology: User Study<br>804 Participants & Feedback"] B --> C["Methodology: Dataset Creation<br>Hybrid User-Intent & Interaction Data"] C --> D["Benchmarking Process<br>11 LLM Services Evaluated"] D --> E["Computational Process<br>LLM-as-Judge Methodology"] E --> F["Key Outcome<br>High Alignment: Pearson r=0.78"] F --> G["Final Contribution<br>UCFE Framework & Benchmark"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes the UCFE (User-Centric Financial Expertise) benchmark, a framework combining human expert judgments with LLM-as-Judge evaluations to assess LLMs on real-world financial tasks.
Source sentence
To address these challenges in financial domain, we propose a novel framework, the User-Centric Financial Expertise Benchmark, designed to eval-uate the ability of LLMs to handle financial tasks in real-world scenarios.
Assumes 2
- LLMs are expected to process information in near real‑time by consolidating unstructured data from multiple sources. model
Source sentence
Financial markets can react instantly to news, making it crucial for LLMs to process information in near real-time by rapidly consoli-dating unstructured, real-time data from multiple sources (Nguyen and Tulabandhula, 2023; Tong et al., 2024).
- The survey of 804 participants accurately captures the needs and intents of the target user groups for benchmark construction. data
Source sentence
Firstly, we conducted a user study involving 804 participants, collecting their feedback on financial tasks. Secondly, based on this feedback, we created our dataset that encompasses a wide range of user intents and interactions.
Data
- UCFE dataset benchmark unknown
Source sentence
This dataset serves as the foundation for benchmarking 11 LLMs services using the LLM-as-Judge methodology. We developed 17 task types tailored to these user profiles, encompassing 330 data points that in-clude multi-round dialogues in both zero-shot and few-shot settings.
Costs
Not applicable
Finds 3
- Benchmark scores correlate strongly with human preferences, achieving a Pearson correlation of 0.78. statistical
Source sentence
Our results show a significant alignment between benchmark scores and human preferences, with a Pearson correlation coefficient of 0.78, confirming the effectiveness of the UCFE dataset and our evaluation approach.
- Financially‑specialized LLMs consistently outperform general‑purpose models on the 17 UCFE tasks. performance figure from elsewhere in the paper
Source sentence
A key finding is the consistent outperformance of financially-specialized LLMs (Tongyi-Finance-LLM as AI Assistant GPT-4o as User User Simulator Select Retrieve Model List Source Information … Hey, AI Assistant, I recently heard something called a balance sheet, but I don’t quite understand what it really is.
- Larger models generally achieve higher scores, though Llama3.1 is an outlier to this scaling trend. performance
Source sentence
LLMs with larger parameters generally outperform smaller ones within the same series, also LLMs from the same backbone model have better results after being trained on financial corpus.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 18 related papers). Coverage is partial, so this is neither replication nor refutation.
2 papers in the archive
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (70)
- A User-Centric Benchmark for Evaluating Large Language Models (2024) — link
- FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models (2023) — link
- InternLM2 Technical Report (2024) — link
- CFGPT: Chinese financial assistant with large language model (2023) — link
- GPT-4 Technical Report (2024) — link
- Using ELO ratings for match result prediction in association football (2010)
- The LDBC Financial Benchmark (2023) — link
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023) — link
and 62 more
- SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applications (2024) — link
- BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark (2023) — link
- FinanceBench: A New Benchmark for Financial Question Answering (2023) — link
- CFGPT: Chinese Financial Assistant with Large Language Model (2023) — link
- CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model (2024) — link
- Palmyra-Fin-70B-32k: a powerful LLM designed for Finance (2024)
- Qwen2.5: A Party of Foundation Models (2024) — link
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024) — link
- XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters (2023) — link
- Llama 3 Model Card (2024) — link
- Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications — → 2408.11878
- FinGPT: Open-Source Financial Large Language Models — → 2306.06031
- A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges — → 2406.11903
- No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks (2024) — link
- Split and Merge: Aligning Position Biases in Large Language Model based Evaluators (2023) — link
- Can Large Language Models be Used to Provide Psychological Counselling? An Analysis of GPT-4-Generated Responses Using Role-play Dialogues (2024) — link
- Bloomberggpt: A large language model for finance (2023) — link
- FinBERT: A large language model for extracting information from financial text (2023)
- Financial statement analysis with large language models — → 2407.17866
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (2024) — link
- Bias and Toxicity in Role-Play Reasoning (2024) — link
- Humans or llms as the judge? a study on judgement biases (2024) — link
- Machine learning: an applied econometric approach (2017)
- Financial markets and growth: An overview (1993)
- More than just noise? Examining the information content of stock microblogs on financial markets (2018)
- Mental accounting and consumer choice (2008)
- The framing of decisions and the psychology of choice (1981)
- A size-of-loss model for the negatively skewed insurance claims data: applications, risk analysis using different methods and statistical forecasting (2024)
- Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
- Comparative Analysis of NLP-Based Models for Company Classification (2024)
- Leveraging vision-language models for granular market change prediction — → 2301.10166
- A survey on large language model (llm) security and privacy: The good, the bad, and the ugly (2024)
- What’s in Your" Safe" Data?: Identifying Benign Data that Breaks Safety (2024) — link
- Challenging the appearance of machine intelligence: Cognitive bias in LLMs and Best Practices for Adoption (2023) — link
- Can LLMs Learn Macroeconomic Narratives from Social Media? (2024) — link
- Generative AI based virtual assistant for reconciliation research (2024)
- Ploutos: Towards interpretable stock movement prediction with financial large language model — → 2403.00782
- Generative AI for Business Strategy: Using Foundation Models to Create Business Strategy Tools (2023) — link
- Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates (2024) — link
- When flue meets flang: Benchmarks and large pre-trained language model for financial domain (2022) — link
- Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi (2024)
- MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024) — link
- Scaling Laws for Neural Language Models (2020) — link
- Observational Scaling Laws and the Predictability of Language Model Performance (2024) — link
- Over-Reasoning and Redundant Calculation of Large Language Models (2024) — link
- Pixiu: A large language model, instruction data and evaluation benchmark for finance (2023) — link
- Large language models and generative AI in finance: an analysis of ChatGPT, Bard, and Bing AI (2023)
- Learning to generate explainable stock predictions using self-reflective large language models — → 2402.03659
- Are ChatGPT and GPT-4 general-purpose solvers for financial text analytics? A study on several typical tasks (2023) — link
- R-judge: Benchmarking safety risk awareness for llm agents (2024) — link
- EUCA: The end-user-centered explainable AI framework (2021) — link
- Exploring the nature and implementation process of user-centric business models (2011)
- Enterprise design thinking: An investigation on user-centered design processes in large corporations (2021)
- Explainable Robo-Advisors: Empirical Investigations to Specify and Evaluate a User-Centric Taxonomy of Explanations in the Financial Domain. (2022)
- A human-centric framework for robotic task learning and optimization (2023)
- Artificial intelligence-enhanced predictive insights for advancing financial inclusion: A human-centric ai-thinking approach (2020)
- Responsible and human centric AI-based insurance advisors (2023)
- User-Centric Explainability in Fintech Applications (2021)
- AI-driven UX/UI design: Empirical research and applications in FinTech (2024)
- Chatbot arena: An open platform for evaluating llms by human preference (2024) — link
- TwinMarket: A Scalable Behavioral and SocialSimulation for Financial Markets (2025) — link
- Econagent: large language model-empowered agents for simulating macroeconomic activities (2024)
8 of 70 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.