Paper: arXiv 2403.15062
Authors: Masanori Hirano
Abstract
With the recent development of large language models (LLMs), models that focus on certain domains and languages have been discussed for their necessity. There is also a growing need for benchmarks to evaluate the performance of current LLMs in each domain. Therefore, in this study, we constructed a benchmark comprising multiple tasks specific to the Japanese and financial domains and performed benchmark measurements on some models. Consequently, we confirmed that GPT-4 is currently outstanding, and that the constructed benchmarks function effectively. According to our analysis, our benchmark can differentiate benchmark scores among models in all performance ranges by combining tasks with different difficulties.
Complexity vs Empirical Score
- Math Complexity: 1.5/10
- Empirical Rigor: 6.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper focuses on benchmark construction and model evaluation using existing datasets and public resources, showing high empirical rigor through data handling and backtest-like evaluations, but the mathematics involved is relatively low-level (mostly classification metrics and data processing).
Research Flowchart
flowchart TD A["Research Goal:<br>Construct Japanese Financial Benchmark for LLMs"] --> B["Methodology:<br>Develop Multi-Task Dataset (QA, Generation, Summarization)"] B --> C["Data & Inputs:<br>Japanese Financial Documents & Financial QA"] C --> D["Computational Process:<br>Evaluate GPT-4 & Existing Models<br>using constructed tasks"] D --> E["Outcome 1:<br>GPT-4 Outperforms Other Models"] D --> F["Outcome 2:<br>Benchmark Effectively Differentiates<br>Model Performance Across Domains"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a new benchmark for evaluating Japanese financial large language models, consisting of five specialized tasks.
Source sentence
We construct a new Japanese financial benchmark for LLMs, comprising the following five benchmark tasks: • chabsa: Sentiment analysis task in the financial field. • cma_basics: Fundamental knowledge questions in securities analysis. • cpa_audit: Tasks on auditing in the Japanese Certified Public Accountant (CPA) exam.
Assumes
No explicit assumptions found in the text.
Data
- chabsa benchmark public
Source sentence
chabsa: Sentiment analysis task in the financial field.
- cma_basics benchmark public
Source sentence
cma_basics: Fundamental knowledge questions in securities analysis.
- cpa_audit benchmark public
Source sentence
cpa_audit: Tasks on auditing in the Japanese Certified Public Accountant (CPA) exam.
- fp2 benchmark public
Source sentence
fp2: Multiple choice questions for 2nd grade Japanese financial planner exam.
- security_sales_1 benchmark public
Source sentence
security_sales_1: Practice exam for the 1st grade Japanese securities broker representative test.
Costs
Not applicable
Finds 2
- GPT‑4 achieved the best performance among the evaluated models on the Japanese financial benchmark. performance
Source sentence
we confirmed that GPT-4 is currently outstanding
- The benchmark can differentiate scores across models in all performance ranges by combining tasks of varying difficulty. performance
Source sentence
our benchmark can differentiate benchmark scores among models in all performance ranges by combining tasks with different difficulties
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 11 related papers). Coverage is partial, so this is neither replication nor refutation.
4 papers in the archive
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (42)
- ChatGPT (2023)
- GPT-4 Technical Report (2023) — link
- A framework for few-shot language model evaluation (2021) — link
- Is it All Hype? ChatGPT’s Performance and Disruptive Potential in the Accounting and Auditing Industries (2023) — link
- Capabilities of GPT-4 on Medical Challenge Problems (2023)
- ChatGPT by OpenAI: The End of Litigation Lawyers? (2023) — link
- ChatGPT Goes to Law School (2023) — link
- Attention Is All You Need (2017)
and 34 more
- PaLM 2 Technical Report (2023)
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023)
- Improving Language Understanding by Generative Pre-Training (2018)
- Language Models are Unsupervised Multitask Learners (2019)
- Language Models are Few-Shot Learners (2020)
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
- Bard (2023)
- Dolly (2023)
- Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (2023)
- LLaMA: Open and Efficient Foundation Language Models (2023)
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022)
- PaLM: Scaling Language Modeling with Pathways (2022)
- BloombergGPT: A Large Language Model for Finance (2023)
- Fin-LLAMA: Efficient Finetuning of Quantized LLMs for Finance (2023)
- FinGPT: Open-Source Financial Large Language Models — → 2306.06031
- Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models — → 2306.12659
- From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models (2023)
- JMedLoRA: Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuning (2023)
- llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology (2023) — link
- JP Language Model Evaluation Harness (2023)
- Gemini: a family of highly capable multimodal models (2023) — link
- Constructing and Analyzing Domain-Specific Language Model for Financial Text Mining (2023) — link
- llm-jp-eval (2024) — link
- Distributed Representations of Words and Phrases and their Compositionality (2013) — link
- Related Stocks Selection with Data Collaboration Using Text Mining (2019) — link
- Deep Contextualized Word Representations (2018) — link
- Bidirectional recurrent neural networks (1997) — link
- Universal Language Model Fine-tuning for Text Classification (2018) — link
- SciBERT: A Pretrained Language Model for Scientific Text (2019)
- Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction (2021)
- Construction and Validation of a Pre-Training and Additional Pre-Training Financial Language Model [in Japanese] (2022) — link
- Don’t stop pretraining: Adapt language models to domains and tasks (2020) — link
- chABSA: Aspect Based Sentiment Analysis dataset in Japanese (2018)
- Can ChatGPT pass the JCPA exam?: Challenge for the short-answer method test on Auditing (2023)
2 of 42 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.