Paper: arXiv 2504.05862

Authors: Takehiro Takayanagi, Kiyoshi Izumi, Javier Sanz-Cruzado, Richard McCreadie, Iadh Ounis

Abstract

Large language model-based agents are becoming increasingly popular as a low-cost mechanism to provide personalized, conversational advice, and have demonstrated impressive capabilities in relatively simple scenarios, such as movie recommendations. But how do these agents perform in complex high-stakes domains, where domain expertise is essential and mistakes carry substantial risk? This paper investigates the effectiveness of LLM-advisors in the finance domain, focusing on three distinct challenges: (1) eliciting user preferences when users themselves may be unsure of their needs, (2) providing personalized guidance for diverse investment preferences, and (3) leveraging advisor personality to build relationships and foster trust. Via a lab-based user study with 64 participants, we show that LLM-advisors often match human advisor performance when eliciting preferences, although they can struggle to resolve conflicting user needs. When providing personalized advice, the LLM was able to positively influence user behavior, but demonstrated clear failure modes. Our results show that accurate preference elicitation is key, otherwise, the LLM-advisor has little impact, or can even direct the investor toward unsuitable assets. More worryingly, users appear insensitive to the quality of advice being given, or worse these can have an inverse relationship. Indeed, users reported a preference for and increased satisfaction as well as emotional trust with LLMs adopting an extroverted persona, even though those agents provided worse advice.

Complexity vs Empirical Score

  • Math Complexity: 2.5/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper focuses on experimental methodology and user behavior analysis rather than advanced mathematical modeling or quantitative trading strategies. Empirical rigor is solid with a structured lab-based user study involving 64 participants, comparing personalized vs. non-personalized advisors and assessing outcomes, though it lacks real-world backtesting or financial data processing.

Research Flowchart

  flowchart TD
  A["Research Goal:<br>How effective are LLM-advisors in high-stakes finance?"] --> B["Methodology:<br>Lab-based user study (n=64 participants)"]
  
  B --> C["Input 1:<br>Preference Elicitation<br>via conversational prompts"]
  B --> D["Input 2:<br>Personalized Advice<br>for diverse investment needs"]
  B --> E["Input 3:<br>Persona Engineering<br>(e.g., extroverted vs. neutral)"]
  
  C & D & E --> F["LLM Computational Process:<br>Conversational Agent +<br>Persona-based reasoning"]
  
  F --> G["Key Findings/Outcomes:"]
  G --> H["1. Preference elicitation matches humans<br>but struggles with conflicts"]
  G --> I["2. Can influence user behavior,<br>but prone to failure modes"]
  G --> J["3. Users prefer extroverted personas<br>despite receiving worse advice"]
  G --> K["4. Accurate elicitation is critical;<br>bad inputs lead to poor outcomes"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper conducts a lab-based user study to evaluate large language model advisors in finance, focusing on preference elicitation, investment personalization, and advisor personality.

Source sentence
6 Conclusion In this paper, we have conducted a lab-based user study to examine how effective large language models are as financial advisors. We focus on three core challenges: preference elicitation, investment personalization, and advisor personality.

Assumes 5

  • The expert‑produced scores for assets constitute a reliable ground‑truth ranking for evaluating LLM advice. model
    Source sentence
    Importantly, as we know the investor profile 𝑖𝑝𝑟𝑒𝑓for each con-versation about an asset 𝑎, we can objectively determine whether 𝑎 is a good investment given 𝑖𝑝𝑟𝑒𝑓, forming a ground truth against which we can compare to the rating provided by our human par-ticipant after their conversation with the LLM-advisor. For each asset 𝑎, a financial expert produced a score between 0 and 3 by manually checking whether
  • Investor profiles are known for each conversation, allowing objective assessment of asset suitability. data
    Source sentence
    Importantly, as we know the investor profile 𝑖𝑝𝑟𝑒𝑓for each con-versation about an asset 𝑎, we can objectively determine whether 𝑎 is a good investment given 𝑖𝑝𝑟𝑒𝑓, forming a ground truth against which we can compare to the rating provided by our human par-ticipant after their conversation with the LLM-advisor.
  • Providing up‑to‑date asset information in the prompt (prices, business summary, recent performance) is sufficient for the LLM to generate accurate advice. execution
    Source sentence
    Baseline Prompt: As we are working with an LLM-advisor and the nature of financial information-seeking is time-sensitive, we need to provide any information that might change over time to the LLM within the prompt. As such, for each asset 𝑎, we pre-prepared a standard asset descriptor block after consulting with a financial expert, containing: • Stock Prices: We collect monthly stock prices from 2023 using Yahoo!
  • Participants can accurately rank assets after advisory discussions, serving as a proxy for decision quality. other
    Source sentence
    (6) Asset Ranking and Feedback: Participants rank all the stocks (four in total) discussed in the advisory session ac-cording to their desire to invest in each.
Show 1 more
  • The LLM‑advisor’s personality can be altered via a context statement without changing its underlying model. model
    Source sentence
    The LLM-advisor is provided with context about the current asset (see Section 3.3), and depending on the experimen-tal scenario, optionally personalization information (step 4 output) and/or a target personality context statement.

Data

  • Yahoo Finance monthly stock prices 2023 · Yahoo Finance market-prices public
    2023
    Source sentence
    As such, for each asset 𝑎, we pre-prepared a standard asset descriptor block after consulting with a financial expert, containing: • Stock Prices: We collect monthly stock prices from 2023 using Yahoo!
  • Yahoo Finance business summaries · Yahoo Finance fundamentals public
    Source sentence
    Finance.2 • Business Summary: We gather each company’s business overview from Yahoo!
  • Seeking Alpha earnings conference call transcripts (Q4 2023) · Seeking Alpha text-news public
    2023 Q4
    Source sentence
    Finance. • Recent Performance and Key Financial Indicators (e.g., EPS): We obtain earnings conference call transcripts3 from Seeking Alpha for the last quarter of 2023.
  • Financial expert asset scores (0‑3) synthetic proprietary
    Source sentence
    For each asset 𝑎, a financial expert produced a score between 0 and 3 by manually checking whether 𝑎satisfied each of the three investment criterial contained in 𝑖𝑝𝑟𝑒𝑓.

Costs

Not applicable The paper reports no trading or execution result.

Finds 5

  • LLM advisors often match human advisor performance in preference elicitation, though they can struggle with conflicting user needs. performance
    Source sentence
    Via a lab-based user study with 64 participants, we show that LLM-advisors often match human advisor performance when eliciting preferences, although they can struggle to resolve conflicting user needs.
  • When preference elicitation is accurate, personalized LLM advice improves investors’ decisions; inaccurate elicitation can lead to unsuitable asset choices. performance
    Source sentence
    When providing personalized advice, the LLM was able to positively influence user behavior, but demonstrated clear failure modes. Second, personalizing LLMs to provide investment advice can improve the decisions made by the investors, but only when the personalized LLM-advisor receives accurate information about the investor’s preferences.
  • Users report higher satisfaction and trust for extroverted LLM advisors even though those advisors provide lower‑quality advice. negative
    Source sentence
    Indeed, users reported a preference for and increased satisfaction as well as emotional trust with LLMs adopting an extroverted persona, even though those agents provided worse advice. More worryingly, when comparing two personalized advisors with ex-troverted and conscientious personalities, we observed that, even though the extroverted advisor provided lower-quality advice, par-ticipants trusted this advisor more
  • LLM advisors achieve elicitation accuracy close to that of an expert human (around 85% average), but can drop to random levels for complex profiles. performance figure from elsewhere in the paper
    Source sentence
    First, our analysis shows that LLMs are effective tools for prefer-ence elicitation through conversation. In a majority of cases, they are capable of obtaining investor’s preferences with an accuracy close to or equivalent to that of an expert human advisor.
  • Personalized LLM advisors lead to better decision outcomes, yet participants often cannot distinguish personalized from non‑personalized systems. performance
    Source sentence
    In the comparison between a non-personalized and a personalized LLM-advisor, although the personalized system led to better decisions, participants were unable to distinguish between the systems.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 10 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 1 paper in the archive (1 by independent authors): Evaluating LLMs in Finance Requires Explicit Bias … (discusses its findings)

Code

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (146)

  1. The patent holder’s dilemma: Buy, sell, or troll? (2007) — link
  2. Deciding equivalances among conjunctive aggregate queries (2007) — link
  3. Special issue: Digital Libraries (1996)
  4. Understanding Policy-Based Networking (2001)
  5. First-Order Dynamic Logic (1979) — link
  6. The title of book one (2007) — link
  7. The title of book two (2008) — link
  8. Achieving application requirements (1990) — link
and 138 more
  1. Statecarts in use: structured analysis and object-orientation (1998) — link
  2. The Art of Computer Programming, Vol. 1: Fundamental Algorithms (3rd. ed.) (1997)
  3. The Art of Computer Programming (1998)
  4. An experiment in bibliographic mark-up: Parsing metadata for XML export (2010) — link
  5. Catch me, if you can: Evading network signatures with web-based polymorphic worms (2007)
  6. Catch me, if you can: Evading network signatures with web-based polymorphic worms (2008)
  7. Catch me, if you can: Evading network signatures with web-based polymorphic worms (2009)
  8. Predicate Path expressions (1979) — link
  9. LOGICS of Programs: AXIOMATICS and DESCRIPTIVE POWER (1978)
  10. Optimal Motion Control of a Ground Vehicle (2003)
  11. Algorithms for Closest-Point Problems (Computational Geometry) (1985)
  12. Introduction to Bayesian Statistics (2001) — link
  13. CLIFFORD: a Maple 11 Package for Clifford Algebra Computations, version 11 (2007) — link
  14. Stats and Analysis (2006) — link
  15. A more perfect union (2008) — link
  16. The fountain of youth (2009)
  17. Solder man (2003) — link
  18. Interview with Bill Kinder: January 13, 2005 (2005) — link
  19. The Enabling of Digital Libraries (2008)
  20. (old) Finding minimum congestion spanning trees (2000) — link
  21. (new) Finding minimum congestion spanning trees (2000) — link
  22. (old) Distributed data source verification in wireless sensor networks (2009) — link
  23. (new) Distributed data source verification in wireless sensor networks (2009) — link
  24. Portalis: using competitive online interactions to support aid initiatives for the homeless (2008) — link
  25. Visual Basic 6: Design, Specification, and Objects with Other (1999)
  26. The Latex Web Companion: Integrating TEX, HTML, and XML (1999)
  27. Vertex Types in Book-Embeddings (1987)
  28. Vertex Types in Book-Embeddings (1987)
  29. CHI ‘08: CHI ‘08 extended abstracts on Human factors in computing systems (2008)
  30. Algorithms for Closest-Point Problems (Computational Geometry) (1985)
  31. IEEE TCSC Executive Committee (2004) — link
  32. Distributed systems (2nd Ed.) (1993)
  33. New Algorithms for Dependency-Directed Backtracking (Master’s thesis) (1986)
  34. New Algorithms for Dependency-Directed Backtracking (Master’s thesis) (1986)
  35. Seminumerical Algorithms (1981)
  36. E-commerce and cultural values (2001) — link
  37. Chapter 9 (2002) — link
  38. The implementation of electronic commerce in SMEs in Singapore (Incoll) (2003) — link
  39. E-commerce and cultural values - (InBook-num-in-chap) (2004) — link
  40. E-commerce and cultural values (Inbook-text-in-chap) (2005) — link
  41. E-commerce and cultural values (Inbook-num chap) (2006) — link
  42. A library-based synthesis methodology for reversible logic (2010)
  43. Synthesis of Reversible Circuit Using Cycle-Based Approach (2010)
  44. Algorithmic Enumeration of Ideal Classes for Quaternion Orders (2010) — link
  45. Chapter II: Notes on data structuring (1972) — link
  46. Transcript of question and answer session (1981) — link
  47. Go to statement considered harmful (1979) — link
  48. Three-dimensional virtual acoustic displays (1992) — link
  49. Managerial expert systems and organizational change: some critical research issues (1987) — link
  50. Simplified Structured COBOL with Microsoft/MicroFocus COBOL (1990)
  51. A multi-radio unification protocol for IEEE 802.11 wireless networks (2004)
  52. Wireless Sensor Networks: A Survey (2002)
  53. A Survey on Wireless Multimedia Sensor Networks (2007)
  54. SSCH : Slotted Seeded Channel Hopping for Capacity Improvement in IEEE 802.11 Ad-Hoc Wireless Networks (2004)
  55. XBOW Sensor Motes Specifications (2008)
  56. Overview of Sensor Networks (2004)
  57. CodeBlue : Sensor Networks for Medical Care (2008)
  58. Investigating Network Architectures for Body Sensor Networks (2007)
  59. Channel-Hopping Multiple Access (2000)
  60. Body Sensor Networks (2008)
  61. Mapping Powerlists onto Hypercubes (1994)
  62. Automatic Parallelization for Distributed-Memory Multiprocessing Systems (1989)
  63. User recovery and reversal in interactive systems (1984)
  64. Generalizing specifications for uniformly implemented loops (1985)
  65. Towards monolingual programming environments (1985)
  66. The \TeX book (1984)
  67. Distributed algorithms for finding centers and medians in networks (1984)
  68. \it \LaTeX : A Document Preparation System (1986)
  69. Program transformations in a denotational setting (1985)
  70. Seminumerical Algorithms (1981)
  71. A high-level approach to computer document formatting (1980)
  72. A multifrequency MAC specially designed for wireless sensor network applications (2010) — link
  73. Institutional members of the \TeX Users Group (2017) — link
  74. acmart— C lass for typesetting publications of ACM (2017) — link
  75. Reasoning About Naming Systems (1993) — link
  76. Babel, a Multilingual Style-Option System for Use with LaTeX’s Standard Document Styles (1991)
  77. A Methodology for Implementing Highly Concurrent Data Objects (1993) — link
  78. Publication quality tables in \LaTeX (2005)
  79. Using the amsthm Package (2015)
  80. R: A Language and Environment for Statistical Computing (2019) — link
  81. UMass Citation Field Extraction Dataset (2013) — link
  82. Citation concept analysis (CCA) —A new form of citation analysis revealing the usefulness of concepts for other researchers illustrated by two exemplary case studies including classic books by Thomas S.~Kuhn and Karl R.~Popper (2019) — link
  83. Learning Soft Linear Constraints with Application to Citation Field Extraction (2014) — link
  84. Maintaining Discrete Probability Distributions Optimally (1993)
  85. Conversational information seeking (2023)
  86. A survey on conversational recommender systems (2021)
  87. A theoretical framework for conversational search (2017)
  88. Towards conversational recommender systems (2016)
  89. Conversational recommender system (2018)
  90. A user-centric evaluation framework for recommender systems (2011)
  91. The effects of personalization and familiarity on trust and adoption of recommendation agents (2006)
  92. When does web-based personalization really work? The distinction between actual personalization and perceived personalization (2016)
  93. Measuring the impact of online personalisation: Past, present and future (2019)
  94. Towards conversational search and recommendation: System ask, user respond (2018)
  95. Soliciting user preferences in conversational recommender systems via usage-related questions (2021)
  96. Coached conversational preference elicitation: A case study in understanding movie preferences (2019)
  97. Intelligent travel chatbot for predictive recommendation in echo platform (2018)
  98. Recognizing users feedback from non-verbal communicative acts in conversational recommender systems (2017)
  99. The effect of preference elicitation methods on the user experience in conversational recommender systems (2025)
  100. Evaluating conversational recommender systems via user simulation (2020)
  101. User evaluation of a conversational recommender system (2005)
  102. An introduction to the five-factor model and its applications. (1992)
  103. P ersona LLM : Investigating the Ability of Large Language Models to Express Personality Traits (2024)
  104. Evaluating and inducing personality in pre-trained language models (2024)
  105. Character-LLM: A Trainable Agent for Role-Playing (2023)
  106. Personality traits in large language models (2023) — link
  107. The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots (2021)
  108. Chatbot personalities matters: improving the user experience of chatbot interfaces (2019)
  109. Impacts of personal characteristics on user trust in conversational recommender systems (2022)
  110. FAR-AI: A Modular Platform for Investment Recommendation in the Financial Domain (2024)
  111. Personalized Stock Recommendation with Investors’ Attention and Contextual Information (2023)
  112. Can ChatGPT Plan Your Retirement?: Generative AI and Financial Advice (2024)
  113. The Evolution of Vanguard Advisor’s Alpha®: From Portfolios to People (2018)
  114. Introduction to recommender systems handbook (2010)
  115. Chatbot Development as a Digital Learning Tool to Increase Students’ Research Knowledge. (2021)
  116. Style Investing (2003)
  117. Value versus growth: The international evidence (1998)
  118. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2023)
  119. An Exploratory Study of Stock Price Movements from Earnings Calls (2022)
  120. Laboratory experiments in information retrieval (2018)
  121. When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks (2011)
  122. What do Users Really Ask Large Language Models? An Initial Log Analysis of Google Bard Interactions in the Wild (2024)
  123. Re-evaluating the Command-and-Control Paradigm in Conversational Search Interactions (2024)
  124. Experimental methods: Eliciting risk preferences (2013)
  125. Risk preference elicitation and financial advice taking (2023)
  126. When ’’ A Helpful Assistant ’’ Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models (2024)
  127. LLMs for Financial Advisement: A Fairness and Efficacy Study in Personal Decision Making (2023)
  128. Revolutionizing Finance with LLMs: An Overview of Applications and Insights (2024) — link
  129. Talking to Multi-Party Conversational Agents in Advisory Services: Command-based vs. Conversational Interactions (2024)
  130. Pen-and-paper Rituals in Service Interaction: Combining High-touch and High-tech in Financial Advisory Encounters (2019)
  131. Conversational robo advisors as surrogates of trust: onboarding experience, firm perception, and consumer financial decision making (2021)
  132. Beyond Turing Test: Can GPT-4 Sway Experts’ Decisions? (2024) — link
  133. Trusting automation: Designing for responsivity and resilience (2023)
  134. Modeling human trust and reliance in AI-assisted decision making: a markovian approach (2023)
  135. Emotional intelligence of large language models (2023)
  136. Recommender systems meet finance: a literature review (2016)
  137. Putting a value on your value: Quantifying Vanguard Adviser’s Alpha in the UK (2020)
  138. Mixed-initiative interaction (1999)

0 of 146 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.