{"paper_id":"2506.08762","title":"EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements","authors":["Issa Sugiura","Takashi Ishida","Taro Makino","Chieko Tazuke","Takanori Nakagawa","Kosuke Nakago","David Ha"],"paper_date":"2025-06-10","paper_type":"dataset-benchmark","paper_url":"https://arxiv.org/pdf/2506.08762v1","page_url":"https://thequant.space/flowcharts/edinet-bench-evaluating-llms-on-complex-financial-tasks-usi/","scores":{"empirical_rigor":8,"math_complexity":2,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper proposes EDINET-Bench, an open-source Japanese financial benchmark for evaluating LLMs on tasks like accounting fraud detection, earnings forecasting, and industry classification.","quote":"We introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate LLMs on challenging tasks such as accounting fraud detection, earnings forecasting, and industry classification.","summary":"Financial analysis presents complex challenges that could leverage large language model (LLM) capabilities. However, the scarcity of challenging financial datasets, particularly for Japanese financial data, impedes academic innovation in financial analytics. As LLMs advance, this lack of accessible…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The training and test splits are company-wise disjoint to avoid overlap.","type":"data","stated":"explicit","quote":"We then split the fraudulent and non-fraudulent samples into training and test sets, ensuring that data from the same company does not overlap between the two sets."},{"assumption":"Labels for fraud and earnings are accurate as derived from EDINET filings.","type":"data","stated":"implied","quote":"For the non-fraudulent reports, we randomly sample 700 companies from all annual reports submitted over the past ten years, excluding those identified as fraudulent."},{"assumption":"Test set contamination from model pre‑training data is negligible.","type":"other","stated":"explicit","quote":"The results show no declining trend for recent years, indicating that test set contamination is unlikely to be affecting our evaluation."}]},"datasets":{"status":"available","items":[{"name":"EDINET-Bench","provider":"Japan’s Electronic Disclosure for Investors’ NETwork (EDINET","kind":"benchmark","access":"public","period":"ten years of annual reports","universe":"Japanese listed companies","quote":"EDINET-Bench is constructed from ten years of annual reports filed by Japanese companies."}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"weekly-monthly"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"State‑of‑the‑art LLMs only marginally outperform logistic regression on binary classification tasks such as fraud detection and earnings forecasting.","kind":"performance","quote":"Our experiments show that even state-of-the-art LLMs struggle in this domain, performing only marginally better than logistic regression in binary classification tasks such as fraud detection and earnings forecasting."},{"finding":"GPT‑5 achieves a ROC‑AUC of 0.65 on earnings forecasting, comparable to prior low‑performance reports.","kind":"predictive-accuracy","quote":"Even GPT-5 achieves only a ROC-AUC of 0.65, aligning with the low forecasting performance reported by Xie et al. (2024)."},{"finding":"Claude 3.5 Sonnet reaches the highest industry prediction accuracy of 0.41 when using full report sections.","kind":"performance","quote":"Claude 3.5 Sonnet achieves state-of-the-art performance with an accuracy score of 0.41 when provided with Summary+BS+CF+PL."}],"reported_performance":null,"out_of_sample":{"method":"holdout","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":20,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":true,"code_url":"https://github.com/SakanaAI/EDINET-Bench","note":null},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:35:40Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}