Paper: arXiv 2403.15062

Authors: Masanori Hirano

Abstract

With the recent development of large language models (LLMs), models that focus on certain domains and languages have been discussed for their necessity. There is also a growing need for benchmarks to evaluate the performance of current LLMs in each domain. Therefore, in this study, we constructed a benchmark comprising multiple tasks specific to the Japanese and financial domains and performed benchmark measurements on some models. Consequently, we confirmed that GPT-4 is currently outstanding, and that the constructed benchmarks function effectively. According to our analysis, our benchmark can differentiate benchmark scores among models in all performance ranges by combining tasks with different difficulties.

Complexity vs Empirical Score

  • Math Complexity: 1.5/10
  • Empirical Rigor: 6.0/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper focuses on benchmark construction and model evaluation using existing datasets and public resources, showing high empirical rigor through data handling and backtest-like evaluations, but the mathematics involved is relatively low-level (mostly classification metrics and data processing).

Research Flowchart

  flowchart TD
  A["Research Goal:<br>Construct Japanese Financial Benchmark for LLMs"] --> B["Methodology:<br>Develop Multi-Task Dataset (QA, Generation, Summarization)"]
  B --> C["Data & Inputs:<br>Japanese Financial Documents & Financial QA"]
  C --> D["Computational Process:<br>Evaluate GPT-4 & Existing Models<br>using constructed tasks"]
  D --> E["Outcome 1:<br>GPT-4 Outperforms Other Models"]
  D --> F["Outcome 2:<br>Benchmark Effectively Differentiates<br>Model Performance Across Domains"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a new benchmark for evaluating Japanese financial large language models, consisting of five specialized tasks.

Source sentence
We construct a new Japanese financial benchmark for LLMs, comprising the following five benchmark tasks: • chabsa: Sentiment analysis task in the financial field. • cma_basics: Fundamental knowledge questions in securities analysis. • cpa_audit: Tasks on auditing in the Japanese Certified Public Accountant (CPA) exam.

Assumes

No explicit assumptions found in the text.

Data

  • chabsa benchmark public
    Source sentence
    chabsa: Sentiment analysis task in the financial field.
  • cma_basics benchmark public
    Source sentence
    cma_basics: Fundamental knowledge questions in securities analysis.
  • cpa_audit benchmark public
    Source sentence
    cpa_audit: Tasks on auditing in the Japanese Certified Public Accountant (CPA) exam.
  • fp2 benchmark public
    Source sentence
    fp2: Multiple choice questions for 2nd grade Japanese financial planner exam.
  • security_sales_1 benchmark public
    Source sentence
    security_sales_1: Practice exam for the 1st grade Japanese securities broker representative test.

Costs

Not applicable The paper reports no trading or execution result.

Finds 2

  • GPT‑4 achieved the best performance among the evaluated models on the Japanese financial benchmark. performance
    Source sentence
    we confirmed that GPT-4 is currently outstanding
  • The benchmark can differentiate scores across models in all performance ranges by combining tasks of varying difficulty. performance
    Source sentence
    our benchmark can differentiate benchmark scores among models in all performance ranges by combining tasks with different difficulties

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 11 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 4 papers in the archive (2 by independent authors)

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (42)

  1. ChatGPT (2023)
  2. GPT-4 Technical Report (2023) — link
  3. A framework for few-shot language model evaluation (2021) — link
  4. Is it All Hype? ChatGPT’s Performance and Disruptive Potential in the Accounting and Auditing Industries (2023) — link
  5. Capabilities of GPT-4 on Medical Challenge Problems (2023)
  6. ChatGPT by OpenAI: The End of Litigation Lawyers? (2023) — link
  7. ChatGPT Goes to Law School (2023) — link
  8. Attention Is All You Need (2017)
and 34 more
  1. PaLM 2 Technical Report (2023)
  2. Llama 2: Open Foundation and Fine-Tuned Chat Models (2023)
  3. Improving Language Understanding by Generative Pre-Training (2018)
  4. Language Models are Unsupervised Multitask Learners (2019)
  5. Language Models are Few-Shot Learners (2020)
  6. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
  7. Bard (2023)
  8. Dolly (2023)
  9. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (2023)
  10. LLaMA: Open and Efficient Foundation Language Models (2023)
  11. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022)
  12. PaLM: Scaling Language Modeling with Pathways (2022)
  13. BloombergGPT: A Large Language Model for Finance (2023)
  14. Fin-LLAMA: Efficient Finetuning of Quantized LLMs for Finance (2023)
  15. FinGPT: Open-Source Financial Large Language Models — → 2306.06031
  16. Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models — → 2306.12659
  17. From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models (2023)
  18. JMedLoRA: Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuning (2023)
  19. llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology (2023) — link
  20. JP Language Model Evaluation Harness (2023)
  21. Gemini: a family of highly capable multimodal models (2023) — link
  22. Constructing and Analyzing Domain-Specific Language Model for Financial Text Mining (2023) — link
  23. llm-jp-eval (2024) — link
  24. Distributed Representations of Words and Phrases and their Compositionality (2013) — link
  25. Related Stocks Selection with Data Collaboration Using Text Mining (2019) — link
  26. Deep Contextualized Word Representations (2018) — link
  27. Bidirectional recurrent neural networks (1997) — link
  28. Universal Language Model Fine-tuning for Text Classification (2018) — link
  29. SciBERT: A Pretrained Language Model for Scientific Text (2019)
  30. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction (2021)
  31. Construction and Validation of a Pre-Training and Additional Pre-Training Financial Language Model [in Japanese] (2022) — link
  32. Don’t stop pretraining: Adapt language models to domains and tasks (2020) — link
  33. chABSA: Aspect Based Sentiment Analysis dataset in Japanese (2018)
  34. Can ChatGPT pass the JCPA exam?: Challenge for the short-answer method test on Auditing (2023)

2 of 42 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.