Paper: arXiv 2506.08762

Authors: Issa Sugiura, Takashi Ishida, Taro Makino, Chieko Tazuke, Takanori Nakagawa, Kosuke Nakago, David Ha

Abstract

Financial analysis presents complex challenges that could leverage large language model (LLM) capabilities. However, the scarcity of challenging financial datasets, particularly for Japanese financial data, impedes academic innovation in financial analytics. As LLMs advance, this lack of accessible research resources increasingly hinders their development and evaluation in this specialized domain. To address this gap, we introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate the performance of LLMs on challenging financial tasks including accounting fraud detection, earnings forecasting, and industry prediction. EDINET-Bench is constructed by downloading annual reports from the past 10 years from Japan’s Electronic Disclosure for Investors’ NETwork (EDINET) and automatically assigning labels corresponding to each evaluation task. Our experiments reveal that even state-of-the-art LLMs struggle, performing only slightly better than logistic regression in binary classification for fraud detection and earnings forecasting. These results highlight significant challenges in applying LLMs to real-world financial applications and underscore the need for domain-specific adaptation. Our dataset, benchmark construction code, and evaluation code is publicly available to facilitate future research in finance with LLMs.

Complexity vs Empirical Score

  • Math Complexity: 2.0/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper focuses on dataset creation, benchmarking, and comparing LLM performance with logistic regression using real-world financial data, with minimal mathematical derivations.

Research Flowchart

  flowchart TD
  A["Research Goal: Evaluate LLMs on complex Japanese financial tasks"] --> B["Methodology: Construct EDINET-Bench"]
  B --> C["Data Source: 10 Years of EDINET Annual Reports"]
  C --> D["Tasks: Fraud Detection, Earnings Forecasting, Industry Prediction"]
  D --> E["Evaluation: SOTA LLMs vs. Baselines"]
  E --> F["Outcome: LLMs struggle vs. simple baselines"]
  F --> G["Conclusion: Need for domain-specific adaptation"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes EDINET-Bench, an open-source Japanese financial benchmark for evaluating LLMs on tasks like accounting fraud detection, earnings forecasting, and industry classification.

Source sentence
We introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate LLMs on challenging tasks such as accounting fraud detection, earnings forecasting, and industry classification.

Assumes 3

  • The training and test splits are company-wise disjoint to avoid overlap. data
    Source sentence
    We then split the fraudulent and non-fraudulent samples into training and test sets, ensuring that data from the same company does not overlap between the two sets.
  • Labels for fraud and earnings are accurate as derived from EDINET filings. data
    Source sentence
    For the non-fraudulent reports, we randomly sample 700 companies from all annual reports submitted over the past ten years, excluding those identified as fraudulent.
  • Test set contamination from model pre‑training data is negligible. other
    Source sentence
    The results show no declining trend for recent years, indicating that test set contamination is unlikely to be affecting our evaluation.

Data

  • EDINET-Bench · Japan’s Electronic Disclosure for Investors’ NETwork (EDINET benchmark public
    ten years of annual reports · Japanese listed companies
    Source sentence
    EDINET-Bench is constructed from ten years of annual reports filed by Japanese companies.

Costs

Not applicable The paper reports no trading or execution result.

Finds 3

  • State‑of‑the‑art LLMs only marginally outperform logistic regression on binary classification tasks such as fraud detection and earnings forecasting. performance
    Source sentence
    Our experiments show that even state-of-the-art LLMs struggle in this domain, performing only marginally better than logistic regression in binary classification tasks such as fraud detection and earnings forecasting.
  • GPT‑5 achieves a ROC‑AUC of 0.65 on earnings forecasting, comparable to prior low‑performance reports. predictive-accuracy
    Source sentence
    Even GPT-5 achieves only a ROC-AUC of 0.65, aligning with the low forecasting performance reported by Xie et al. (2024).
  • Claude 3.5 Sonnet reaches the highest industry prediction accuracy of 0.41 when using full report sections. performance
    Source sentence
    Claude 3.5 Sonnet achieves state-of-the-art performance with an accuracy score of 0.41 when provided with Summary+BS+CF+PL.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 20 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

https://github.com/SakanaAI/EDINET-Bench — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (88)

  1. Automated design of agentic systems (2025)
  2. Predicting industry sectors from financial statements: An illustration of machine learning in accounting research (2022)
  3. Financial statement analysis and the prediction of stock returns (1989)
  4. Predicting future earnings changes using machine learning and detailed financial data (2022)
  5. GDP (current US$) (2025) — link
  6. LiveBench: A Challenging, Contamination-Limited LLM Benchmark (2025)
  7. An Evolved Universal Transformer Memory (2024) — link
  8. Construction of domain-specified japanese large language model for finance through continual pre-training — → 2404.10555
and 80 more
  1. C onv F in QA : Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering (2022) — link
  2. F in QA : A Dataset of Numerical Reasoning over Financial Data (2021) — link
  3. Industry Momentum Strategy Based on Text Mining in the Japanese Stock Market (2022) — link
  4. 不正会計の早期発見に関する海外調査・研究報告書 (2014)
  5. Research on Early Detection of Inappropriate Accounting in Response to Recent Changes in Economic Environment (2018) — link
  6. Trends in Accounting Fraud among Listed Companies (2024) — link
  7. Introducing O3 and O4-mini (2025) — link
  8. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning (2025) — link
  9. Imagenet: A large-scale hierarchical image database (2009)
  10. GLUE: A multi-task benchmark and analysis platform for natural language understanding (2018) — link
  11. FAMMA : A Benchmark for Financial Domain Multilingual Multimodal Question Answering (2025) — link
  12. Can C hat GPT pass the JCPA exam? C hallenge for the short-answer method test on auditing (2023)
  13. Officebench: Benchmarking language agents across multiple applications for office automation (2024) — link
  14. Insightbench: Evaluating business analytics agents through multi-step insight generation (2025)
  15. AFlow: Automating Agentic Workflow Generation (2025)
  16. CoARiJ: Corpus of Annual Reports in Japan (2020) — link
  17. EDGAR - CORPUS : Billions of Tokens Make The World Go Round (2021) — link
  18. Predicting future earnings changes using machine learning and detailed financial data (2022)
  19. A Machine Learning Approach to Industry Classification in Financial Markets (2023)
  20. Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning (2025) — link
  21. Rethinking benchmark and contamination for language models with rephrased samples (2023) — link
  22. Data Contamination Can Cross Language Barriers (2024) — link
  23. Claude 3.7 Sonnet and Claude Code (2025) — link
  24. Learning to Reason with LLMs (2024) — link
  25. DeepSeek-V3 Technical Report (2025) — link
  26. Measuring Mathematical Problem Solving With the MATH Dataset (2021) — link
  27. Let’s verify step by step (2024)
  28. FinRobot: an open-source AI agent platform for financial applications using large language models — → 2405.14767
  29. Bloomberg GPT : A Large Language Model for Finance (2023) — link
  30. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (2024)
  31. Large Language Models as Optimizers (2024)
  32. Automated Design of Agentic Systems (2025)
  33. Fin Ben : A Holistic Financial Benchmark for Large Language Models (2024) — link
  34. Finance Bench : A New Benchmark for Financial Question Answering (2023) — link
  35. Construction of a Japanese Financial Benchmark for Large Language Models — → 2403.15062
  36. Using machine learning to detect and forecast accounting fraud (2019)
  37. Financial Statement Analysis with Large Language Models — → 2407.17866
  38. LIMO: Less is More for Reasoning (2025) — link
  39. GPT-4 T echnical R eport (2024) — link
  40. Claude 3.5 Sonnet (2024) — link
  41. Our next-generation model: Gemini 1.5 (2024) — link
  42. PIXIU : A Large Language Model, Instruction Data and Evaluation Benchmark for Finance (2023) — link
  43. The Detection of Earnings Manipulation (1999) — link
  44. Predicting accounting fraud: Evidence from Japan (2016)
  45. Financial statement fraud detection: An analysis of statistical and machine learning algorithms (2011)
  46. The Llama 3 Herd of Models (2024) — link
  47. Scikit-learn: Machine Learning in P ython (2011)
  48. F in E val: A C hinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models (2025) — link
  49. Predicting material accounting misstatements (2011)
  50. Intelligent financial fraud detection: A comprehensive review (2016) — link
  51. Fin GPT : Open-Source Financial Large Language Models (2023)
  52. Health B ench: Evaluating Large Language Models Towards Improved Human Health (2025) — link
  53. PaperBench : Evaluating AI \textquoteright s Ability to Replicate AI Research (2025) — link
  54. Financial Statement Analysis and Earnings Forecasting (2018) — link
  55. A comparison of dividend, cash flow, and earnings approaches to equity valuation (1998)
  56. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) — link
  57. Proving Test Set Contamination in Black-Box Language Models (2024) — link
  58. Measuring short-form factuality in large language models (2024) — link
  59. Evaluating Large Language Models Trained on Code (2021) — link
  60. SWE -bench: Can Language Models Resolve Real-world Github Issues? (2024) — link
  61. WebArena : A Realistic Web Environment for Building Autonomous Agents (2024) — link
  62. OpenHands : An Open Platform for AI Software Developers as Generalist Agents (2025) — link
  63. Large Language Models in Finance (FinLLMs) (2025) — link
  64. Humanity’s Last Exam (2025) — link
  65. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — link
  66. Survey and Perspectives on Large Language Models in J apanese Financial Sector (2025) — link
  67. pdfminer (2016)
  68. Polars (2021)
  69. SWE -agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) — link
  70. Time Travel in LLM s: Tracing Data Contamination in Large Language Models (2024) — link
  71. MLE -bench: Evaluating Machine Learning Agents on Machine Learning Engineering (2025) — link
  72. Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023) — link
  73. FinanceMATH : Knowledge-Intensive Math Reasoning in Finance Domains (2024) — link
  74. Automating Financial Statement Audits with Large Language Models (2024) — link
  75. Kimi K2 : Open Agentic Intelligence (2025) — link
  76. XGBoost : A Scalable Tree Boosting System (2016) — link
  77. Random Forests (2001) — link
  78. The regression analysis of binary sequences (1958)
  79. Introducing SWE -bench Verified (2024) — link
  80. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (2025) — link

4 of 88 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.