Paper: arXiv 2404.10555
Authors: Masanori Hirano, Kentaro Imajo
Abstract
Large language models (LLMs) are now widely used in various fields, including finance. However, Japanese financial-specific LLMs have not been proposed yet. Hence, this study aims to construct a Japanese financial-specific LLM through continual pre-training. Before tuning, we constructed Japanese financial-focused datasets for continual pre-training. As a base model, we employed a Japanese LLM that achieved state-of-the-art performance on Japanese financial benchmarks among the 10-billion-class parameter models. After continual pre-training using the datasets and the base model, the tuned model performed better than the original model on the Japanese financial benchmarks. Moreover, the outputs comparison results reveal that the tuned model’s outputs tend to be better than the original model’s outputs in terms of the quality and length of the answers. These findings indicate that domain-specific continual pre-training is also effective for LLMs. The tuned model is publicly available on Hugging Face.
Complexity vs Empirical Score
- Math Complexity: 1.5/10
- Empirical Rigor: 8.5/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper uses advanced ML concepts like transformer architectures and continual pre-training, but the focus is on implementation details (data prep, training hyperparameters) and empirical results (benchmarks, public model release).
Research Flowchart
flowchart TD
A["Research Goal: Construct<br>Japanese Financial LLM"] --> B["Dataset Construction"]
B --> C["Select Base Model<br>Japanese LLM 10B"]
C --> D["Continual Pre-training<br>on Financial Data"]
D --> E{"Evaluation"}
E --> F["Key Findings: Outcomes"]
subgraph F ["Key Findings: Outcomes"]
F1["Benchmark Improvement<br>Better than original model"]
F2["Output Quality<br>Higher quality & length"]
F3["Public Release<br>Hugging Face"]
end
B -->|"Input: Japanese<br>Financial Texts"| D
C -->|"Input: 10B Parameter<br>Japanese LLM"| D
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes constructing a Japanese financial‑specific large language model by continual pre‑training on domain‑focused data.
Source sentence
Hence, this study aims to construct a Japanese financial-specific LLM through continual pre-training.
Assumes 4
- Continual pre‑training is effective for domain‑specific tuning. model
Source sentence
Therefore, we employed the continual pre-training method for domain-specific tuning in this study.
- The base model is state‑of‑the‑art for Japanese financial benchmarks. model
Source sentence
we need to employ the state-of-the-art model as a base model.
- The constructed datasets are legally usable for commercial purposes in Japan as of April 2024. data
Source sentence
The datasets are currently clear to use for commercial purposes under Japanese law as of April 2024.
- Benchmark scores reflect model performance on Japanese financial tasks. statistical
Source sentence
The tuned model achieved better performance than the original model in all tasks.
Data
- Japanese financial‑focused datasets text-news public
Source sentence
The crawled articles mainly include the following types of documents: • Speeches, Press Conferences, and Talks of Officers of the Bank of Japan • Minutes of the Monetary Policy Meetings of the Bank of Japan • Reports, glossaries, and company profiles from multiple financial institutions • Financial-related documents ex
Costs
Not applicable
Finds 4
- The tuned model’s overall benchmark score is 0.4716, which is +0.0381 higher than the original model’s score of 0.4335. performance
Source sentence
The overall score of the tuned model is 0.4716, which is +0.0381 higher than the original model’s score (0.4335).
- The tuned model outperforms the original model on every individual benchmark task. performance
Source sentence
The tuned model achieved better performance than the original model in all tasks.
- During continual pre‑training, the loss curve showed no spikes and saturated as the learning rate decayed linearly to zero. statistical
Source sentence
In our tuning, no loss spikes were observed. The loss curve was also saturated as the learning rate decayed linearly to 0.
- Outputs from the tuned model tend to be better than those from the original model in terms of quality and length of answers. other
Source sentence
According to those examples, the tuned model’s outputs tend to be better than the original model’s outputs in terms of the quality and length of the answers.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 15 related papers). Coverage is partial, so this is neither replication nor refutation.
1 paper in the archive
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (50)
- ChatGPT (2023)
- GPT-4 Technical Report (2023) — link
- A framework for few-shot language model evaluation (2021) — link
- Is it All Hype? ChatGPT’s Performance and Disruptive Potential in the Accounting and Auditing Industries (2023) — link
- Capabilities of GPT-4 on Medical Challenge Problems (2023)
- ChatGPT by OpenAI: The End of Litigation Lawyers? (2023) — link
- ChatGPT Goes to Law School (2023) — link
- Attention Is All You Need (2017)
and 42 more
- PaLM 2 Technical Report (2023)
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023)
- Improving Language Understanding by Generative Pre-Training (2018)
- Language Models are Unsupervised Multitask Learners (2019)
- Language Models are Few-Shot Learners (2020)
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
- Bard (2023)
- Dolly (2023)
- Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (2023)
- LLaMA: Open and Efficient Foundation Language Models (2023)
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022)
- PaLM: Scaling Language Modeling with Pathways (2022)
- BloombergGPT: A Large Language Model for Finance (2023)
- Fin-LLAMA: Efficient Finetuning of Quantized LLMs for Finance (2023)
- FinGPT: Open-Source Financial Large Language Models — → 2306.06031
- Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models — → 2306.12659
- From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models (2023)
- JMedLoRA: Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuning (2023)
- llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology (2023) — link
- JP Language Model Evaluation Harness (2023)
- Gemini: a family of highly capable multimodal models (2023) — link
- Constructing and Analyzing Domain-Specific Language Model for Financial Text Mining (2023) — link
- llm-jp-eval (2024) — link
- Distributed Representations of Words and Phrases and their Compositionality (2013)
- Related Stocks Selection with Data Collaboration Using Text Mining (2019) — link
- Deep Contextualized Word Representations (2018) — link
- Bidirectional recurrent neural networks (1997) — link
- Universal Language Model Fine-tuning for Text Classification (2018) — link
- SciBERT: A Pretrained Language Model for Scientific Text (2019)
- Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction (2021)
- Construction and Validation of a Pre-Training and Additional Pre-Training Financial Language Model [in Japanese] (2022) — link
- Don’t stop pretraining: Adapt language models to domains and tasks (2020) — link
- Construction of a Japanese Financial Benchmark for Large Language Models — → 2403.15062
- Lora: Low-rank adaptation of large language models (2021) — link
- Finetuned language models are zero-shot learners (2021) — link
- Deep reinforcement learning from human preferences (2017)
- Direct preference optimization: Your language model is secretly a reward model (2024)
- Lima: Less is more for alignment (2024)
- Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters (2020)
- Accelerate: Training and inference at scale made simple, efficient and adaptable. (2022)
- Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling (2024) — link
- Can ChatGPT pass the JCPA exam?: Challenge for the short-answer method test on Auditing (2023)
3 of 50 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.