Paper: arXiv 2404.10555

Authors: Masanori Hirano, Kentaro Imajo

Abstract

Large language models (LLMs) are now widely used in various fields, including finance. However, Japanese financial-specific LLMs have not been proposed yet. Hence, this study aims to construct a Japanese financial-specific LLM through continual pre-training. Before tuning, we constructed Japanese financial-focused datasets for continual pre-training. As a base model, we employed a Japanese LLM that achieved state-of-the-art performance on Japanese financial benchmarks among the 10-billion-class parameter models. After continual pre-training using the datasets and the base model, the tuned model performed better than the original model on the Japanese financial benchmarks. Moreover, the outputs comparison results reveal that the tuned model’s outputs tend to be better than the original model’s outputs in terms of the quality and length of the answers. These findings indicate that domain-specific continual pre-training is also effective for LLMs. The tuned model is publicly available on Hugging Face.

Complexity vs Empirical Score

  • Math Complexity: 1.5/10
  • Empirical Rigor: 8.5/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper uses advanced ML concepts like transformer architectures and continual pre-training, but the focus is on implementation details (data prep, training hyperparameters) and empirical results (benchmarks, public model release).

Research Flowchart

  flowchart TD
  A["Research Goal: Construct<br>Japanese Financial LLM"] --> B["Dataset Construction"]
  B --> C["Select Base Model<br>Japanese LLM 10B"]
  C --> D["Continual Pre-training<br>on Financial Data"]
  D --> E{"Evaluation"}
  E --> F["Key Findings: Outcomes"]
  
  subgraph F ["Key Findings: Outcomes"]
      F1["Benchmark Improvement<br>Better than original model"]
      F2["Output Quality<br>Higher quality & length"]
      F3["Public Release<br>Hugging Face"]
  end
  
  B -->|"Input: Japanese<br>Financial Texts"| D
  C -->|"Input: 10B Parameter<br>Japanese LLM"| D

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes constructing a Japanese financial‑specific large language model by continual pre‑training on domain‑focused data.

Source sentence
Hence, this study aims to construct a Japanese financial-specific LLM through continual pre-training.

Assumes 4

  • Continual pre‑training is effective for domain‑specific tuning. model
    Source sentence
    Therefore, we employed the continual pre-training method for domain-specific tuning in this study.
  • The base model is state‑of‑the‑art for Japanese financial benchmarks. model
    Source sentence
    we need to employ the state-of-the-art model as a base model.
  • The constructed datasets are legally usable for commercial purposes in Japan as of April 2024. data
    Source sentence
    The datasets are currently clear to use for commercial purposes under Japanese law as of April 2024.
  • Benchmark scores reflect model performance on Japanese financial tasks. statistical
    Source sentence
    The tuned model achieved better performance than the original model in all tasks.

Data

  • Japanese financial‑focused datasets · crawled articles from the Internet text-news public
    Source sentence
    The crawled articles mainly include the following types of documents: • Speeches, Press Conferences, and Talks of Officers of the Bank of Japan • Minutes of the Monetary Policy Meetings of the Bank of Japan • Reports, glossaries, and company profiles from multiple financial institutions • Financial-related documents ex

Costs

Not applicable The paper reports no trading or execution result.

Finds 4

  • The tuned model’s overall benchmark score is 0.4716, which is +0.0381 higher than the original model’s score of 0.4335. performance
    Source sentence
    The overall score of the tuned model is 0.4716, which is +0.0381 higher than the original model’s score (0.4335).
  • The tuned model outperforms the original model on every individual benchmark task. performance
    Source sentence
    The tuned model achieved better performance than the original model in all tasks.
  • During continual pre‑training, the loss curve showed no spikes and saturated as the learning rate decayed linearly to zero. statistical
    Source sentence
    In our tuning, no loss spikes were observed. The loss curve was also saturated as the learning rate decayed linearly to 0.
  • Outputs from the tuned model tend to be better than those from the original model in terms of quality and length of answers. other
    Source sentence
    According to those examples, the tuned model’s outputs tend to be better than the original model’s outputs in terms of the quality and length of the answers.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 15 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 1 paper in the archive (1 by independent authors)

Code

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (50)

  1. ChatGPT (2023)
  2. GPT-4 Technical Report (2023) — link
  3. A framework for few-shot language model evaluation (2021) — link
  4. Is it All Hype? ChatGPT’s Performance and Disruptive Potential in the Accounting and Auditing Industries (2023) — link
  5. Capabilities of GPT-4 on Medical Challenge Problems (2023)
  6. ChatGPT by OpenAI: The End of Litigation Lawyers? (2023) — link
  7. ChatGPT Goes to Law School (2023) — link
  8. Attention Is All You Need (2017)
and 42 more
  1. PaLM 2 Technical Report (2023)
  2. Llama 2: Open Foundation and Fine-Tuned Chat Models (2023)
  3. Improving Language Understanding by Generative Pre-Training (2018)
  4. Language Models are Unsupervised Multitask Learners (2019)
  5. Language Models are Few-Shot Learners (2020)
  6. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
  7. Bard (2023)
  8. Dolly (2023)
  9. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (2023)
  10. LLaMA: Open and Efficient Foundation Language Models (2023)
  11. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022)
  12. PaLM: Scaling Language Modeling with Pathways (2022)
  13. BloombergGPT: A Large Language Model for Finance (2023)
  14. Fin-LLAMA: Efficient Finetuning of Quantized LLMs for Finance (2023)
  15. FinGPT: Open-Source Financial Large Language Models — → 2306.06031
  16. Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models — → 2306.12659
  17. From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models (2023)
  18. JMedLoRA: Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuning (2023)
  19. llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology (2023) — link
  20. JP Language Model Evaluation Harness (2023)
  21. Gemini: a family of highly capable multimodal models (2023) — link
  22. Constructing and Analyzing Domain-Specific Language Model for Financial Text Mining (2023) — link
  23. llm-jp-eval (2024) — link
  24. Distributed Representations of Words and Phrases and their Compositionality (2013)
  25. Related Stocks Selection with Data Collaboration Using Text Mining (2019) — link
  26. Deep Contextualized Word Representations (2018) — link
  27. Bidirectional recurrent neural networks (1997) — link
  28. Universal Language Model Fine-tuning for Text Classification (2018) — link
  29. SciBERT: A Pretrained Language Model for Scientific Text (2019)
  30. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction (2021)
  31. Construction and Validation of a Pre-Training and Additional Pre-Training Financial Language Model [in Japanese] (2022) — link
  32. Don’t stop pretraining: Adapt language models to domains and tasks (2020) — link
  33. Construction of a Japanese Financial Benchmark for Large Language Models — → 2403.15062
  34. Lora: Low-rank adaptation of large language models (2021) — link
  35. Finetuned language models are zero-shot learners (2021) — link
  36. Deep reinforcement learning from human preferences (2017)
  37. Direct preference optimization: Your language model is secretly a reward model (2024)
  38. Lima: Less is more for alignment (2024)
  39. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters (2020)
  40. Accelerate: Training and inference at scale made simple, efficient and adaptable. (2022)
  41. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling (2024) — link
  42. Can ChatGPT pass the JCPA exam?: Challenge for the short-answer method test on Auditing (2023)

3 of 50 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.