{"paper_id":"2506.07315","title":"Towards Competent AI for Fundamental Analysis in Finance: A Benchmark Dataset and Evaluation","authors":["Zonghan Wu","Congyuan Zou","Junlin Wang","Chenhan Wang","Hangjing Yang","Yilei Shao"],"paper_date":"2025-05-22","paper_type":"dataset-benchmark","paper_url":"https://arxiv.org/pdf/2506.07315v2","page_url":"https://thequant.space/flowcharts/towards-competent-ai-for-fundamental-analysis-in-finance-a/","scores":{"empirical_rigor":8.5,"math_complexity":2,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper introduces FinAR-Bench, a task‑oriented benchmark dataset for evaluating LLMs on financial statement analysis.","quote":"In this paper, we propose FinAR‑Bench, a solid benchmark dataset focusing on financial statement analysis, a core competence of fundamental analysis.","summary":"Generative AI, particularly large language models (LLMs), is beginning to transform the financial industry by automating tasks and helping to make sense of complex financial information. One especially promising use case is the automatic creation of fundamental analysis reports, which are essential…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"Financial statement analysis can be decomposed into three measurable sub‑tasks: information extraction, indicator calculation, and logical reasoning.","type":"model","stated":"implied","quote":"we break this task into three measurable steps: extracting key information, calculating financial indicators, and applying logical reasoning."}]},"datasets":{"status":"available","items":[{"name":"FinAR‑Bench dataset (100 Shanghai Stock Exchange companies, FY 2023)","provider":null,"kind":"fundamentals","access":"unknown","period":"2023","universe":"100 companies listed on the Shanghai Stock Exchange","quote":"We curate a dataset containing one hundred companies listed on the Shanghai Stock Exchange for the fiscal year 2023."}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"weekly-monthly"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"LLMs achieve high recall (>95%) on information extraction even with 32 items, but small models drop 10‑20 points.","kind":"performance","quote":"In the fact extraction task, large-sized LLMs maintain over 95% recall even at size 32, showing strong robustness. In contrast, small-sized LLMs suffer 10 to 20 percentage point drops."},{"finding":"Indicator computation is more sensitive to task size; top models lose 5‑10 recall points as the number of items increases.","kind":"performance","quote":"to task size. Even top-performing models show 5 to 10 point drops."},{"finding":"Recall improves with numeric tolerance; for GPT‑4o recall rises from 34.38% (θ=0) to 91.47% (θ=1.0).","kind":"performance","quote":"GPT‑4o 34.38 63.69 75.18 80.90 86.67 90.71 91.41 91.47"},{"finding":"Overall LLM ranking shows GPT‑o1 (634.3 wins) and DeepSeek‑r1 (624.5 wins) leading, while Llama‑3.1‑405b‑instruct lags far behind.","kind":"performance","quote":"As shown in Table 2, GPT‑o1 and DeepSeek‑r1 leads the ranking with 634.3 and 624.5 wins, respectively."},{"finding":"In logical reasoning, GPT‑o1 has fewer logic errors than DeepSeek‑r1 but both models perform poorly in analytical depth and financial insight.","kind":"negative","quote":"GPT‑o1 has fewer logic errors than DeepSeek‑r1 and both models perform poorly in terms of analytical depth and financial insight."}],"reported_performance":null,"out_of_sample":{"method":"unclear","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":30,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":true,"code_url":"https://github.com/SAIFS-AIHub/FinAR-Bench","note":null},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:37:15Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}