Paper: arXiv 2409.19854
Authors: Masanori Hirano, Kentaro Imajo
Abstract
This paper proposes a novel method for constructing instruction-tuned large language models (LLMs) for finance without instruction data. Traditionally, developing such domain-specific LLMs has been resource-intensive, requiring a large dataset and significant computational power for continual pretraining and instruction tuning. Our study proposes a simpler approach that combines domain-specific continual pretraining with model merging. Given that general-purpose pretrained LLMs and their instruction-tuned LLMs are often publicly available, they can be leveraged to obtain the necessary instruction task vector. By merging this with a domain-specific pretrained vector, we can effectively create instruction-tuned LLMs for finance without additional instruction data. Our process involves two steps: first, we perform continual pretraining on financial data; second, we merge the instruction-tuned vector with the domain-specific pretrained vector. Our experiments demonstrate the successful construction of instruction-tuned LLMs for finance. One major advantage of our method is that the instruction-tuned and domain-specific pretrained vectors are nearly independent. This independence makes our approach highly effective. The Japanese financial instruction-tuned LLMs we developed in this study are available at https://huggingface.co/pfnet/nekomata-14b-pfn-qfin-inst-merge.
Complexity vs Empirical Score
- Math Complexity: 2.0/10
- Empirical Rigor: 6.0/10 (capped from 8.5: no out-of-sample test; see penalty flags)
- Quadrant: Street Traders — empirical and applied, lower math complexity (effective empirical rigor capped from 8.5)
Why this score: The math complexity is low, as the paper focuses on a high-level methodology (model merging) without heavy mathematical derivations. The empirical rigor is high, evidenced by the construction of a specific Japanese financial corpus, execution of continual pretraining and model merging experiments, and the release of the resulting models on Hugging Face for reproducibility.
Research Flowchart
flowchart TD A["Research Goal<br>Instruction-tuned LLM for Finance<br>without Instruction Data"] --> B["Step 1: Continual Pretraining<br>Pretrain Base LLM on Financial Text"] B --> C["Step 2: Model Merging<br>Merge Domain LLM + Instruction Vector"] C --> D["Key Finding 1<br>Independence of Vectors<br>Effective without Joint Tuning"] C --> E["Key Finding 2<br>Successful Financial LLM<br>Released publicly on Hugging Face"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a method that builds instruction‑tuned financial LLMs by first doing domain‑specific continual pretraining and then merging the resulting model with a publicly available instruction‑tuned LLM.
Source sentence
Our study proposes a simpler approach that combines domain-specific continual pretraining with model merging.
Assumes 1
- Instruction support and domain‑specific knowledge are independent in task arithmetic. model
Source sentence
For the success of the model merging, we assumed that instruction support and domain-specific knowledge are independent in task arithmetic.
Data
- Japanese financial corpus text-news public
Source sentence
The crawled articles mainly included the following types of documents: • speeches and press conferences of officers of the Bank of Japan • minutes of the monetary policy meetings of the Bank of Japan • reports, glossaries, and company profiles from multiple financial institutions • financial documents extracted from Wi
Costs
Not applicable
Finds 3
- The domain‑specific instruction‑tuned LLM achieved the best performance on benchmarks, though the gap to general‑purpose instruction‑tuned LLMs was small. performance
Source sentence
the best performance. However, the difference between the domain-specific and general-purpose instruction-tuned LLMs is not great.
- The proposed continual pretraining and model‑merging pipeline successfully constructs an instruction‑tuned LLM for finance without any instruction data. performance
Source sentence
In this study, we successfully constructed the instruction-tuned LLM for finance without instruction data using continual pretraining and model merging.
- Weight similarity analysis shows low cosine similarity between domain‑specific and instruction task‑arithmetic vectors, supporting the independence assumption. theoretical
Source sentence
Fig. 4. Weight Similarities between the Domain-specific and Instruction Task Arithmetic Vectors of the Merged Model
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 11 related papers). Coverage is partial, so this is neither replication nor refutation.
1 paper in the archive
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (62)
- ChatGPT (2023)
- GPT-4 Technical Report (2023) — link
- A framework for few-shot language model evaluation (2021) — link
- Is it All Hype? ChatGPT’s Performance and Disruptive Potential in the Accounting and Auditing Industries (2023) — link
- Capabilities of GPT-4 on Medical Challenge Problems (2023)
- ChatGPT by OpenAI: The End of Litigation Lawyers? (2023) — link
- ChatGPT Goes to Law School (2023) — link
- Attention Is All You Need (2017)
and 54 more
- PaLM 2 Technical Report (2023)
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023)
- Improving Language Understanding by Generative Pre-Training (2018)
- Language Models are Unsupervised Multitask Learners (2019)
- Language Models are Few-Shot Learners (2020)
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
- Bard (2023)
- Dolly (2023)
- Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (2023)
- LLaMA: Open and Efficient Foundation Language Models (2023)
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022)
- PaLM: Scaling Language Modeling with Pathways (2022)
- BloombergGPT: A Large Language Model for Finance (2023)
- Fin-LLAMA: Efficient Finetuning of Quantized LLMs for Finance (2023)
- FinGPT: Open-Source Financial Large Language Models — → 2306.06031
- Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models — → 2306.12659
- From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models (2023)
- JMedLoRA: Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuning (2023)
- llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology (2023) — link
- JP Language Model Evaluation Harness (2023)
- Gemini: a family of highly capable multimodal models (2023) — link
- Constructing and Analyzing Domain-Specific Language Model for Financial Text Mining (2023) — link
- llm-jp-eval (2024) — link
- Distributed Representations of Words and Phrases and their Compositionality (2013)
- Related Stocks Selection with Data Collaboration Using Text Mining (2019) — link
- Deep Contextualized Word Representations (2018) — link
- Bidirectional recurrent neural networks (1997) — link
- Universal Language Model Fine-tuning for Text Classification (2018) — link
- SciBERT: A Pretrained Language Model for Scientific Text (2019)
- Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction (2021)
- Construction and Validation of a Pre-Training and Additional Pre-Training Financial Language Model [in Japanese] (2022) — link
- Don’t stop pretraining: Adapt language models to domains and tasks (2020) — link
- Lora: Low-rank adaptation of large language models (2021) — link
- Finetuned language models are zero-shot learners (2021) — link
- Deep reinforcement learning from human preferences (2017)
- Direct preference optimization: Your language model is secretly a reward model (2024)
- Lima: Less is more for alignment (2024)
- Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters (2020)
- Accelerate: Training and inference at scale made simple, efficient and adaptable. (2022)
- Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling (2024) — link
- Can ChatGPT pass the JCPA exam?: Challenge for the short-answer method test on Auditing (2023)
- Deep model fusion: A survey (2023) — link
- Editing models with task arithmetic (2022) — link
- Evolutionary optimization of model merging recipes (2024) — link
- FinBERT: A large language model for extracting information from financial text (2023)
- Finbert: A pre-trained financial language representation model for financial text mining (2021)
- LLaMA-2-Econ: Enhancing Title Generation, Abstract Classification, and Academic Q&A in Economic Research (2024)
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022)
- Ties-merging: Resolving interference when merging models (2024)
- Towards comprehensive and efficient post safety alignment of large language models via safety patching (2024) — link
- Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet (2024)
- Construction of a Japanese Financial Benchmark for Large Language Models — → 2403.15062
- Judging llm-as-a-judge with mt-bench and chatbot arena (2024)
- pfmt-bench-fin-ja: Preferred Multi-turn Benchmark for Finance in Japanese (2024) — link
3 of 62 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.