Paper: arXiv 2506.08762
Authors: Issa Sugiura, Takashi Ishida, Taro Makino, Chieko Tazuke, Takanori Nakagawa, Kosuke Nakago, David Ha
Abstract
Financial analysis presents complex challenges that could leverage large language model (LLM) capabilities. However, the scarcity of challenging financial datasets, particularly for Japanese financial data, impedes academic innovation in financial analytics. As LLMs advance, this lack of accessible research resources increasingly hinders their development and evaluation in this specialized domain. To address this gap, we introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate the performance of LLMs on challenging financial tasks including accounting fraud detection, earnings forecasting, and industry prediction. EDINET-Bench is constructed by downloading annual reports from the past 10 years from Japan’s Electronic Disclosure for Investors’ NETwork (EDINET) and automatically assigning labels corresponding to each evaluation task. Our experiments reveal that even state-of-the-art LLMs struggle, performing only slightly better than logistic regression in binary classification for fraud detection and earnings forecasting. These results highlight significant challenges in applying LLMs to real-world financial applications and underscore the need for domain-specific adaptation. Our dataset, benchmark construction code, and evaluation code is publicly available to facilitate future research in finance with LLMs.
Complexity vs Empirical Score
- Math Complexity: 2.0/10
- Empirical Rigor: 8.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper focuses on dataset creation, benchmarking, and comparing LLM performance with logistic regression using real-world financial data, with minimal mathematical derivations.
Research Flowchart
flowchart TD A["Research Goal: Evaluate LLMs on complex Japanese financial tasks"] --> B["Methodology: Construct EDINET-Bench"] B --> C["Data Source: 10 Years of EDINET Annual Reports"] C --> D["Tasks: Fraud Detection, Earnings Forecasting, Industry Prediction"] D --> E["Evaluation: SOTA LLMs vs. Baselines"] E --> F["Outcome: LLMs struggle vs. simple baselines"] F --> G["Conclusion: Need for domain-specific adaptation"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes EDINET-Bench, an open-source Japanese financial benchmark for evaluating LLMs on tasks like accounting fraud detection, earnings forecasting, and industry classification.
Source sentence
We introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate LLMs on challenging tasks such as accounting fraud detection, earnings forecasting, and industry classification.
Assumes 3
- The training and test splits are company-wise disjoint to avoid overlap. data
Source sentence
We then split the fraudulent and non-fraudulent samples into training and test sets, ensuring that data from the same company does not overlap between the two sets.
- Labels for fraud and earnings are accurate as derived from EDINET filings. data
Source sentence
For the non-fraudulent reports, we randomly sample 700 companies from all annual reports submitted over the past ten years, excluding those identified as fraudulent.
- Test set contamination from model pre‑training data is negligible. other
Source sentence
The results show no declining trend for recent years, indicating that test set contamination is unlikely to be affecting our evaluation.
Data
- EDINET-Bench benchmark public
Source sentence
EDINET-Bench is constructed from ten years of annual reports filed by Japanese companies.
Costs
Not applicable
Finds 3
- State‑of‑the‑art LLMs only marginally outperform logistic regression on binary classification tasks such as fraud detection and earnings forecasting. performance
Source sentence
Our experiments show that even state-of-the-art LLMs struggle in this domain, performing only marginally better than logistic regression in binary classification tasks such as fraud detection and earnings forecasting.
- GPT‑5 achieves a ROC‑AUC of 0.65 on earnings forecasting, comparable to prior low‑performance reports. predictive-accuracy
Source sentence
Even GPT-5 achieves only a ROC-AUC of 0.65, aligning with the low forecasting performance reported by Xie et al. (2024).
- Claude 3.5 Sonnet reaches the highest industry prediction accuracy of 0.41 when using full report sections. performance
Source sentence
Claude 3.5 Sonnet achieves state-of-the-art performance with an accuracy score of 0.41 when provided with Summary+BS+CF+PL.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 20 related papers). Coverage is partial, so this is neither replication nor refutation.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (88)
- Automated design of agentic systems (2025)
- Predicting industry sectors from financial statements: An illustration of machine learning in accounting research (2022)
- Financial statement analysis and the prediction of stock returns (1989)
- Predicting future earnings changes using machine learning and detailed financial data (2022)
- GDP (current US$) (2025) — link
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark (2025)
- An Evolved Universal Transformer Memory (2024) — link
- Construction of domain-specified japanese large language model for finance through continual pre-training — → 2404.10555
and 80 more
- C onv F in QA : Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering (2022) — link
- F in QA : A Dataset of Numerical Reasoning over Financial Data (2021) — link
- Industry Momentum Strategy Based on Text Mining in the Japanese Stock Market (2022) — link
- 不正会計の早期発見に関する海外調査・研究報告書 (2014)
- Research on Early Detection of Inappropriate Accounting in Response to Recent Changes in Economic Environment (2018) — link
- Trends in Accounting Fraud among Listed Companies (2024) — link
- Introducing O3 and O4-mini (2025) — link
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning (2025) — link
- Imagenet: A large-scale hierarchical image database (2009)
- GLUE: A multi-task benchmark and analysis platform for natural language understanding (2018) — link
- FAMMA : A Benchmark for Financial Domain Multilingual Multimodal Question Answering (2025) — link
- Can C hat GPT pass the JCPA exam? C hallenge for the short-answer method test on auditing (2023)
- Officebench: Benchmarking language agents across multiple applications for office automation (2024) — link
- Insightbench: Evaluating business analytics agents through multi-step insight generation (2025)
- AFlow: Automating Agentic Workflow Generation (2025)
- CoARiJ: Corpus of Annual Reports in Japan (2020) — link
- EDGAR - CORPUS : Billions of Tokens Make The World Go Round (2021) — link
- Predicting future earnings changes using machine learning and detailed financial data (2022)
- A Machine Learning Approach to Industry Classification in Financial Markets (2023)
- Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning (2025) — link
- Rethinking benchmark and contamination for language models with rephrased samples (2023) — link
- Data Contamination Can Cross Language Barriers (2024) — link
- Claude 3.7 Sonnet and Claude Code (2025) — link
- Learning to Reason with LLMs (2024) — link
- DeepSeek-V3 Technical Report (2025) — link
- Measuring Mathematical Problem Solving With the MATH Dataset (2021) — link
- Let’s verify step by step (2024)
- FinRobot: an open-source AI agent platform for financial applications using large language models — → 2405.14767
- Bloomberg GPT : A Large Language Model for Finance (2023) — link
- Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (2024)
- Large Language Models as Optimizers (2024)
- Automated Design of Agentic Systems (2025)
- Fin Ben : A Holistic Financial Benchmark for Large Language Models (2024) — link
- Finance Bench : A New Benchmark for Financial Question Answering (2023) — link
- Construction of a Japanese Financial Benchmark for Large Language Models — → 2403.15062
- Using machine learning to detect and forecast accounting fraud (2019)
- Financial Statement Analysis with Large Language Models — → 2407.17866
- LIMO: Less is More for Reasoning (2025) — link
- GPT-4 T echnical R eport (2024) — link
- Claude 3.5 Sonnet (2024) — link
- Our next-generation model: Gemini 1.5 (2024) — link
- PIXIU : A Large Language Model, Instruction Data and Evaluation Benchmark for Finance (2023) — link
- The Detection of Earnings Manipulation (1999) — link
- Predicting accounting fraud: Evidence from Japan (2016)
- Financial statement fraud detection: An analysis of statistical and machine learning algorithms (2011)
- The Llama 3 Herd of Models (2024) — link
- Scikit-learn: Machine Learning in P ython (2011)
- F in E val: A C hinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models (2025) — link
- Predicting material accounting misstatements (2011)
- Intelligent financial fraud detection: A comprehensive review (2016) — link
- Fin GPT : Open-Source Financial Large Language Models (2023)
- Health B ench: Evaluating Large Language Models Towards Improved Human Health (2025) — link
- PaperBench : Evaluating AI \textquoteright s Ability to Replicate AI Research (2025) — link
- Financial Statement Analysis and Earnings Forecasting (2018) — link
- A comparison of dividend, cash flow, and earnings approaches to equity valuation (1998)
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) — link
- Proving Test Set Contamination in Black-Box Language Models (2024) — link
- Measuring short-form factuality in large language models (2024) — link
- Evaluating Large Language Models Trained on Code (2021) — link
- SWE -bench: Can Language Models Resolve Real-world Github Issues? (2024) — link
- WebArena : A Realistic Web Environment for Building Autonomous Agents (2024) — link
- OpenHands : An Open Platform for AI Software Developers as Generalist Agents (2025) — link
- Large Language Models in Finance (FinLLMs) (2025) — link
- Humanity’s Last Exam (2025) — link
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — link
- Survey and Perspectives on Large Language Models in J apanese Financial Sector (2025) — link
- pdfminer (2016)
- Polars (2021)
- SWE -agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) — link
- Time Travel in LLM s: Tracing Data Contamination in Large Language Models (2024) — link
- MLE -bench: Evaluating Machine Learning Agents on Machine Learning Engineering (2025) — link
- Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023) — link
- FinanceMATH : Knowledge-Intensive Math Reasoning in Finance Domains (2024) — link
- Automating Financial Statement Audits with Large Language Models (2024) — link
- Kimi K2 : Open Agentic Intelligence (2025) — link
- XGBoost : A Scalable Tree Boosting System (2016) — link
- Random Forests (2001) — link
- The regression analysis of binary sequences (1958)
- Introducing SWE -bench Verified (2024) — link
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (2025) — link
4 of 88 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.