Paper: arXiv 2409.13749

Authors: Neel Rajani, Lilli Kiessling, Aleksandr Ogaltsov, Claus Lang

Abstract

Although powerful, current cutting-edge LLMs may not fulfil the needs of highly specialised sectors. We introduce KodeXv0.1, a family of large language models that outclass GPT-4 in financial question answering. We utilise the base variants of Llama 3.1 8B and 70B and adapt them to the financial domain through a custom training regime. To this end, we collect and process a large number of publicly available financial documents such as earnings calls and business reports. These are used to generate a high-quality, synthetic dataset consisting of Context-Question-Answer triplets which closely mirror real-world financial tasks. Using the train split of this dataset, we perform RAG-aware 4bit LoRA instruction tuning runs of Llama 3.1 base variants to produce KodeX-8Bv0.1 and KodeX-70Bv0.1. We then complete extensive model evaluations using FinanceBench, FinQABench and the withheld test split of our dataset. Our results show that KodeX-8Bv0.1 is more reliable in financial contexts than cutting-edge instruct models in the same parameter regime, surpassing them by up to 9.24%. In addition, it is even capable of outperforming state-of-the-art proprietary models such as GPT-4 by up to 7.07%. KodeX-70Bv0.1 represents a further improvement upon this, exceeding GPT-4’s performance on every tested benchmark.

Complexity vs Empirical Score

  • Math Complexity: 2.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper primarily focuses on data engineering and training infrastructure for LLMs, with minimal advanced mathematical derivations. It demonstrates high empirical rigor through a multi-stage synthetic dataset pipeline, extensive benchmarking against established financial QA benchmarks, and clear comparative performance metrics.

Research Flowchart

  flowchart TD
  A["Research Goal: Adapt LLMs for Specialized Financial QA"] --> B["Data Collection & Processing"]
  B --> C["Synthetic Dataset Generation<br>(Context-Q-A Triplets)"]
  B --> D["Base Model Selection<br>Llama 3.1 8B & 70B"]
  C --> E["Fine-Tuning & Training<br>RAG-aware 4bit LoRA"]
  D --> E
  E --> F["Model Output: KodeX-8Bv0.1 & 70Bv0.1"]
  F --> G["Key Findings"]
  
  G --> G1["KodeX-8Bv0.1: Outperforms GPT-4 by up to 7.07%"]
  G --> G2["KodeX-70Bv0.1: Exceeds GPT-4 on every benchmark"]
  G --> G3["Domain adaptation yields SOTA results for specialized tasks"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes KodeXv0.1, a family of financial large language models fine‑tuned on synthetic data to outperform GPT‑4 on financial QA.

Source sentence
We introduce KodeXv0.1, a family of large language models that outclass GPT-4 in financial question answering.

Assumes

No explicit assumptions found in the text.

Data

  • Synthetic Context‑Question‑Answer dataset · publicly available financial documents (earnings calls, busi synthetic public
    822 documents from over 40 companies listed on major indices such as the MSCI World and Eu
    Source sentence
    A total of 822 documents from over 40 companies were utilized to generate the training dataset.

Costs

Not applicable The paper reports no trading or execution result.

Finds 3

  • KodeX‑8Bv0.1 exceeds other instruct variants in the same parameter regime by up to 9.27 percentage points and beats GPT‑4 by 1.01% on the withheld test set. performance
    Source sentence
    The score of KodeX-8Bv0.1 exceeds that of instruct variants of Mistral and Llama 3.1 in the same parameter regime by up to 9.27 percentage points, and exceeds that of Llama 3.1-70B-instruct and GPT-4 by 3.78% and 1.01%, respectively.
  • KodeX‑70Bv0.1 outperforms GPT‑4 on FinanceBench by 1.93 percentage points, setting state‑of‑the‑art results across all tested benchmarks. performance
    Source sentence
    KodeX-70Bv0.1, which even outperforms GPT-4 by 1.93 percentage points.
  • Across all benchmarks (FinanceBench, FinQABench, and a withheld test split), KodeXv0.1 models achieve the highest scores, establishing new state‑of‑the‑art performance. performance
    Source sentence
    Our KodeXv0.1 models set the state of the art in all benchmarks that we tested.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 10 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (33)

  1. The Theory of Parsing, Translation and Compiling (1972)
  2. Judging LLM-as-a-judge with MT-bench and Chatbot Arena (2024)
  3. OLMo: Accelerating the Science of Language Models (2024) — link
  4. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023) — link
  5. The Llama 3 Herd of Models (2024) — link
  6. Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2 (2023) — link
  7. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) — link
  8. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation (2024) — link
and 25 more
  1. Experiment Tracking with Weights and Biases (2020) — link
  2. TRL: Transformer Reinforcement Learning (2020)
  3. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods (2022)
  4. Llm. int8 (): 8-bit matrix multiplication for transformers at scale (2022) — link
  5. LoRA: Low-Rank Adaptation of Large Language Models (2021) — link
  6. Self-Consistency Improves Chain of Thought Reasoning in Language Models (2023) — link
  7. Empowering financial futures: Large language models in the modern financial landscape (2024)
  8. How Multilingual is Multilingual LLM? (2023) — link
  9. FinanceBench: A New Benchmark for Financial Question Answering (2023) — link
  10. Training language models to follow instructions with human feedback (2022) — link
  11. Mistral 7B (2023) — link
  12. Qwen2 Technical Report (2024) — link
  13. Stanford Alpaca: An Instruction-following LLaMA model (2023)
  14. Self-Instruct: Aligning Language Models with Self-Generated Instructions (2023) — link
  15. Instruction Tuning With Loss Over Instructions (2024) — link
  16. BloombergGPT: A Large Language Model for Finance (2023) — link
  17. ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering (2022) — link
  18. Lighthouzai/finqabench (2024) — link
  19. FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models (2024) — link
  20. Instruction Fine-Tuning: Does Prompt Loss Matter? (2024) — link
  21. Language Models are Unsupervised Multitask Learners (2019)
  22. Language Models are Few-Shot Learners (2020) — link
  23. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2024) — link
  24. ORPO: Monolithic Preference Optimization without Reference Model (2024) — link
  25. Nemotron-4 340B Technical Report (2024) — link

0 of 33 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.