Paper: arXiv 2409.19854

Authors: Masanori Hirano, Kentaro Imajo

Abstract

This paper proposes a novel method for constructing instruction-tuned large language models (LLMs) for finance without instruction data. Traditionally, developing such domain-specific LLMs has been resource-intensive, requiring a large dataset and significant computational power for continual pretraining and instruction tuning. Our study proposes a simpler approach that combines domain-specific continual pretraining with model merging. Given that general-purpose pretrained LLMs and their instruction-tuned LLMs are often publicly available, they can be leveraged to obtain the necessary instruction task vector. By merging this with a domain-specific pretrained vector, we can effectively create instruction-tuned LLMs for finance without additional instruction data. Our process involves two steps: first, we perform continual pretraining on financial data; second, we merge the instruction-tuned vector with the domain-specific pretrained vector. Our experiments demonstrate the successful construction of instruction-tuned LLMs for finance. One major advantage of our method is that the instruction-tuned and domain-specific pretrained vectors are nearly independent. This independence makes our approach highly effective. The Japanese financial instruction-tuned LLMs we developed in this study are available at https://huggingface.co/pfnet/nekomata-14b-pfn-qfin-inst-merge.

Complexity vs Empirical Score

  • Math Complexity: 2.0/10
  • Empirical Rigor: 6.0/10 (capped from 8.5: no out-of-sample test; see penalty flags)
  • Quadrant: Street Traders — empirical and applied, lower math complexity (effective empirical rigor capped from 8.5)

Why this score: The math complexity is low, as the paper focuses on a high-level methodology (model merging) without heavy mathematical derivations. The empirical rigor is high, evidenced by the construction of a specific Japanese financial corpus, execution of continual pretraining and model merging experiments, and the release of the resulting models on Hugging Face for reproducibility.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Instruction-tuned LLM for Finance<br>without Instruction Data"] --> B["Step 1: Continual Pretraining<br>Pretrain Base LLM on Financial Text"]
  B --> C["Step 2: Model Merging<br>Merge Domain LLM + Instruction Vector"]
  C --> D["Key Finding 1<br>Independence of Vectors<br>Effective without Joint Tuning"]
  C --> E["Key Finding 2<br>Successful Financial LLM<br>Released publicly on Hugging Face"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a method that builds instruction‑tuned financial LLMs by first doing domain‑specific continual pretraining and then merging the resulting model with a publicly available instruction‑tuned LLM.

Source sentence
Our study proposes a simpler approach that combines domain-specific continual pretraining with model merging.

Assumes 1

  • Instruction support and domain‑specific knowledge are independent in task arithmetic. model
    Source sentence
    For the success of the model merging, we assumed that instruction support and domain-specific knowledge are independent in task arithmetic.

Data

  • Japanese financial corpus text-news public
    Source sentence
    The crawled articles mainly included the following types of documents: • speeches and press conferences of officers of the Bank of Japan • minutes of the monetary policy meetings of the Bank of Japan • reports, glossaries, and company profiles from multiple financial institutions • financial documents extracted from Wi

Costs

Not applicable The paper reports no trading or execution result.

Finds 3

  • The domain‑specific instruction‑tuned LLM achieved the best performance on benchmarks, though the gap to general‑purpose instruction‑tuned LLMs was small. performance
    Source sentence
    the best performance. However, the difference between the domain-specific and general-purpose instruction-tuned LLMs is not great.
  • The proposed continual pretraining and model‑merging pipeline successfully constructs an instruction‑tuned LLM for finance without any instruction data. performance
    Source sentence
    In this study, we successfully constructed the instruction-tuned LLM for finance without instruction data using continual pretraining and model merging.
  • Weight similarity analysis shows low cosine similarity between domain‑specific and instruction task‑arithmetic vectors, supporting the independence assumption. theoretical
    Source sentence
    Fig. 4. Weight Similarities between the Domain-specific and Instruction Task Arithmetic Vectors of the Merged Model

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 11 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 1 paper in the archive (1 by independent authors)

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (62)

  1. ChatGPT (2023)
  2. GPT-4 Technical Report (2023) — link
  3. A framework for few-shot language model evaluation (2021) — link
  4. Is it All Hype? ChatGPT’s Performance and Disruptive Potential in the Accounting and Auditing Industries (2023) — link
  5. Capabilities of GPT-4 on Medical Challenge Problems (2023)
  6. ChatGPT by OpenAI: The End of Litigation Lawyers? (2023) — link
  7. ChatGPT Goes to Law School (2023) — link
  8. Attention Is All You Need (2017)
and 54 more
  1. PaLM 2 Technical Report (2023)
  2. Llama 2: Open Foundation and Fine-Tuned Chat Models (2023)
  3. Improving Language Understanding by Generative Pre-Training (2018)
  4. Language Models are Unsupervised Multitask Learners (2019)
  5. Language Models are Few-Shot Learners (2020)
  6. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
  7. Bard (2023)
  8. Dolly (2023)
  9. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (2023)
  10. LLaMA: Open and Efficient Foundation Language Models (2023)
  11. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022)
  12. PaLM: Scaling Language Modeling with Pathways (2022)
  13. BloombergGPT: A Large Language Model for Finance (2023)
  14. Fin-LLAMA: Efficient Finetuning of Quantized LLMs for Finance (2023)
  15. FinGPT: Open-Source Financial Large Language Models — → 2306.06031
  16. Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models — → 2306.12659
  17. From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models (2023)
  18. JMedLoRA: Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuning (2023)
  19. llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology (2023) — link
  20. JP Language Model Evaluation Harness (2023)
  21. Gemini: a family of highly capable multimodal models (2023) — link
  22. Constructing and Analyzing Domain-Specific Language Model for Financial Text Mining (2023) — link
  23. llm-jp-eval (2024) — link
  24. Distributed Representations of Words and Phrases and their Compositionality (2013)
  25. Related Stocks Selection with Data Collaboration Using Text Mining (2019) — link
  26. Deep Contextualized Word Representations (2018) — link
  27. Bidirectional recurrent neural networks (1997) — link
  28. Universal Language Model Fine-tuning for Text Classification (2018) — link
  29. SciBERT: A Pretrained Language Model for Scientific Text (2019)
  30. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction (2021)
  31. Construction and Validation of a Pre-Training and Additional Pre-Training Financial Language Model [in Japanese] (2022) — link
  32. Don’t stop pretraining: Adapt language models to domains and tasks (2020) — link
  33. Lora: Low-rank adaptation of large language models (2021) — link
  34. Finetuned language models are zero-shot learners (2021) — link
  35. Deep reinforcement learning from human preferences (2017)
  36. Direct preference optimization: Your language model is secretly a reward model (2024)
  37. Lima: Less is more for alignment (2024)
  38. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters (2020)
  39. Accelerate: Training and inference at scale made simple, efficient and adaptable. (2022)
  40. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling (2024) — link
  41. Can ChatGPT pass the JCPA exam?: Challenge for the short-answer method test on Auditing (2023)
  42. Deep model fusion: A survey (2023) — link
  43. Editing models with task arithmetic (2022) — link
  44. Evolutionary optimization of model merging recipes (2024) — link
  45. FinBERT: A large language model for extracting information from financial text (2023)
  46. Finbert: A pre-trained financial language representation model for financial text mining (2021)
  47. LLaMA-2-Econ: Enhancing Title Generation, Abstract Classification, and Academic Q&A in Economic Research (2024)
  48. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022)
  49. Ties-merging: Resolving interference when merging models (2024)
  50. Towards comprehensive and efficient post safety alignment of large language models via safety patching (2024) — link
  51. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet (2024)
  52. Construction of a Japanese Financial Benchmark for Large Language Models — → 2403.15062
  53. Judging llm-as-a-judge with mt-bench and chatbot arena (2024)
  54. pfmt-bench-fin-ja: Preferred Multi-turn Benchmark for Finance in Japanese (2024) — link

3 of 62 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.