{"paper_id":"2504.05862","title":"Are Generative AI Agents Effective Personalized Financial Advisors?","authors":["Takehiro Takayanagi","Kiyoshi Izumi","Javier Sanz-Cruzado","Richard McCreadie","Iadh Ounis"],"paper_date":"2025-04-08","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2504.05862v2","page_url":"https://thequant.space/flowcharts/are-generative-ai-agents-effective-personalized-financial-ad/","scores":{"empirical_rigor":6,"math_complexity":2.5,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper conducts a lab-based user study to evaluate large language model advisors in finance, focusing on preference elicitation, investment personalization, and advisor personality.","quote":"6 Conclusion In this paper, we have conducted a lab-based user study to examine how effective large language models are as financial advisors. We focus on three core challenges: preference elicitation, investment personalization, and advisor personality.","summary":"Large language model-based agents are becoming increasingly popular as a low-cost mechanism to provide personalized, conversational advice, and have demonstrated impressive capabilities in relatively simple scenarios, such as movie recommendations. But how do these agents perform in complex…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The expert‑produced scores for assets constitute a reliable ground‑truth ranking for evaluating LLM advice.","type":"model","stated":"explicit","quote":"Importantly, as we know the investor profile 𝑖𝑝𝑟𝑒𝑓for each con-versation about an asset 𝑎, we can objectively determine whether 𝑎 is a good investment given 𝑖𝑝𝑟𝑒𝑓, forming a ground truth against which we can compare to the rating provided by our human par-ticipant after their conversation with the LLM-advisor. For each asset 𝑎, a financial expert produced a score between 0 and 3 by manually checking whether","support":"full"},{"assumption":"Investor profiles are known for each conversation, allowing objective assessment of asset suitability.","type":"data","stated":"explicit","quote":"Importantly, as we know the investor profile 𝑖𝑝𝑟𝑒𝑓for each con-versation about an asset 𝑎, we can objectively determine whether 𝑎 is a good investment given 𝑖𝑝𝑟𝑒𝑓, forming a ground truth against which we can compare to the rating provided by our human par-ticipant after their conversation with the LLM-advisor.","support":"full"},{"assumption":"Providing up‑to‑date asset information in the prompt (prices, business summary, recent performance) is sufficient for the LLM to generate accurate advice.","type":"execution","stated":"explicit","quote":"Baseline Prompt: As we are working with an LLM-advisor and the nature of financial information-seeking is time-sensitive, we need to provide any information that might change over time to the LLM within the prompt. As such, for each asset 𝑎, we pre-prepared a standard asset descriptor block after consulting with a financial expert, containing: • Stock Prices: We collect monthly stock prices from 2023 using Yahoo!","support":"full"},{"assumption":"Participants can accurately rank assets after advisory discussions, serving as a proxy for decision quality.","type":"other","stated":"implied","quote":"(6) Asset Ranking and Feedback: Participants rank all the stocks (four in total) discussed in the advisory session ac-cording to their desire to invest in each.","support":"full"},{"assumption":"The LLM‑advisor’s personality can be altered via a context statement without changing its underlying model.","type":"model","stated":"implied","quote":"The LLM-advisor is provided with context about the current asset (see Section 3.3), and depending on the experimen-tal scenario, optionally personalization information (step 4 output) and/or a target personality context statement.","support":"full"}]},"datasets":{"status":"available","items":[{"name":"Yahoo Finance monthly stock prices 2023","provider":"Yahoo Finance","kind":"market-prices","access":"public","period":"2023","universe":null,"quote":"As such, for each asset 𝑎, we pre-prepared a standard asset descriptor block after consulting with a financial expert, containing: • Stock Prices: We collect monthly stock prices from 2023 using Yahoo!","support":"full","known_id":"yahoo-finance"},{"name":"Yahoo Finance business summaries","provider":"Yahoo Finance","kind":"fundamentals","access":"public","period":null,"universe":null,"quote":"Finance.2 • Business Summary: We gather each company’s business overview from Yahoo!","support":"full","known_id":"yahoo-finance"},{"name":"Seeking Alpha earnings conference call transcripts (Q4 2023)","provider":"Seeking Alpha","kind":"text-news","access":"public","period":"2023 Q4","universe":null,"quote":"Finance. • Recent Performance and Key Financial Indicators (e.g., EPS): We obtain earnings conference call transcripts3 from Seeking Alpha for the last quarter of 2023.","support":"full"},{"name":"Financial expert asset scores (0‑3)","provider":null,"kind":"synthetic","access":"proprietary","period":null,"universe":null,"quote":"For each asset 𝑎, a financial expert produced a score between 0 and 3 by manually checking whether 𝑎satisfied each of the three investment criterial contained in 𝑖𝑝𝑟𝑒𝑓.","support":"full"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"none"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"unclear","findings":[{"finding":"LLM advisors often match human advisor performance in preference elicitation, though they can struggle with conflicting user needs.","kind":"performance","quote":"Via a lab-based user study with 64 participants, we show that LLM-advisors often match human advisor performance when eliciting preferences, although they can struggle to resolve conflicting user needs.","support":"full"},{"finding":"When preference elicitation is accurate, personalized LLM advice improves investors’ decisions; inaccurate elicitation can lead to unsuitable asset choices.","kind":"performance","quote":"When providing personalized advice, the LLM was able to positively influence user behavior, but demonstrated clear failure modes. Second, personalizing LLMs to provide investment advice can improve the decisions made by the investors, but only when the personalized LLM-advisor receives accurate information about the investor’s preferences.","support":"full"},{"finding":"Users report higher satisfaction and trust for extroverted LLM advisors even though those advisors provide lower‑quality advice.","kind":"negative","quote":"Indeed, users reported a preference for and increased satisfaction as well as emotional trust with LLMs adopting an extroverted persona, even though those agents provided worse advice. More worryingly, when comparing two personalized advisors with ex-troverted and conscientious personalities, we observed that, even though the extroverted advisor provided lower-quality advice, par-ticipants trusted this advisor more","support":"full"},{"finding":"LLM advisors achieve elicitation accuracy close to that of an expert human (around 85% average), but can drop to random levels for complex profiles.","kind":"performance","quote":"First, our analysis shows that LLMs are effective tools for prefer-ence elicitation through conversation. In a majority of cases, they are capable of obtaining investor’s preferences with an accuracy close to or equivalent to that of an expert human advisor.","support":"partial"},{"finding":"Personalized LLM advisors lead to better decision outcomes, yet participants often cannot distinguish personalized from non‑personalized systems.","kind":"performance","quote":"In the comparison between a non-personalized and a personalized LLM-advisor, although the personalized system led to better decisions, participants were unable to distinguish between the systems.","support":"full"}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[],"contradicts":[],"qualifies":[],"compared_with":10,"citations":{"cited_by_in_corpus":1,"by_independent_authors":1,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":1,"mentions":0,"unclassified":0},"independent_papers":[{"paper":{"paper_id":"2602.14233","title":"Evaluating LLMs in Finance Requires Explicit Bias Consideration","paper_date":"2026-02-15","authors":["Yaxuan Kong","Hoyoung Lee","Yoontae Hwang","Alejandro Lopez-Lira","Bradford Levy","Dhagash Mehta"],"empirical_rigor":6.5,"page_url":"https://thequant.space/flowcharts/evaluating-llms-in-finance-requires-explicit-bias-consideration/"},"how":"discusses_findings","quote":"This misalignment leads to critical risk where users prioritize persona over accuracy [CITED]."}],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":true,"code_url":"https://github.com/TTsamurai/LLMAdvisor_supplementary","note":null},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":2,"extracted_at":"2026-11-14T19:51:03Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}