Paper: arXiv 2409.13749
Authors: Neel Rajani, Lilli Kiessling, Aleksandr Ogaltsov, Claus Lang
Abstract
Although powerful, current cutting-edge LLMs may not fulfil the needs of highly specialised sectors. We introduce KodeXv0.1, a family of large language models that outclass GPT-4 in financial question answering. We utilise the base variants of Llama 3.1 8B and 70B and adapt them to the financial domain through a custom training regime. To this end, we collect and process a large number of publicly available financial documents such as earnings calls and business reports. These are used to generate a high-quality, synthetic dataset consisting of Context-Question-Answer triplets which closely mirror real-world financial tasks. Using the train split of this dataset, we perform RAG-aware 4bit LoRA instruction tuning runs of Llama 3.1 base variants to produce KodeX-8Bv0.1 and KodeX-70Bv0.1. We then complete extensive model evaluations using FinanceBench, FinQABench and the withheld test split of our dataset. Our results show that KodeX-8Bv0.1 is more reliable in financial contexts than cutting-edge instruct models in the same parameter regime, surpassing them by up to 9.24%. In addition, it is even capable of outperforming state-of-the-art proprietary models such as GPT-4 by up to 7.07%. KodeX-70Bv0.1 represents a further improvement upon this, exceeding GPT-4’s performance on every tested benchmark.
Complexity vs Empirical Score
- Math Complexity: 2.5/10
- Empirical Rigor: 8.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper primarily focuses on data engineering and training infrastructure for LLMs, with minimal advanced mathematical derivations. It demonstrates high empirical rigor through a multi-stage synthetic dataset pipeline, extensive benchmarking against established financial QA benchmarks, and clear comparative performance metrics.
Research Flowchart
flowchart TD A["Research Goal: Adapt LLMs for Specialized Financial QA"] --> B["Data Collection & Processing"] B --> C["Synthetic Dataset Generation<br>(Context-Q-A Triplets)"] B --> D["Base Model Selection<br>Llama 3.1 8B & 70B"] C --> E["Fine-Tuning & Training<br>RAG-aware 4bit LoRA"] D --> E E --> F["Model Output: KodeX-8Bv0.1 & 70Bv0.1"] F --> G["Key Findings"] G --> G1["KodeX-8Bv0.1: Outperforms GPT-4 by up to 7.07%"] G --> G2["KodeX-70Bv0.1: Exceeds GPT-4 on every benchmark"] G --> G3["Domain adaptation yields SOTA results for specialized tasks"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes KodeXv0.1, a family of financial large language models fine‑tuned on synthetic data to outperform GPT‑4 on financial QA.
Source sentence
We introduce KodeXv0.1, a family of large language models that outclass GPT-4 in financial question answering.
Assumes
No explicit assumptions found in the text.
Data
- Synthetic Context‑Question‑Answer dataset synthetic public
Source sentence
A total of 822 documents from over 40 companies were utilized to generate the training dataset.
Costs
Not applicable
Finds 3
- KodeX‑8Bv0.1 exceeds other instruct variants in the same parameter regime by up to 9.27 percentage points and beats GPT‑4 by 1.01% on the withheld test set. performance
Source sentence
The score of KodeX-8Bv0.1 exceeds that of instruct variants of Mistral and Llama 3.1 in the same parameter regime by up to 9.27 percentage points, and exceeds that of Llama 3.1-70B-instruct and GPT-4 by 3.78% and 1.01%, respectively.
- KodeX‑70Bv0.1 outperforms GPT‑4 on FinanceBench by 1.93 percentage points, setting state‑of‑the‑art results across all tested benchmarks. performance
Source sentence
KodeX-70Bv0.1, which even outperforms GPT-4 by 1.93 percentage points.
- Across all benchmarks (FinanceBench, FinQABench, and a withheld test split), KodeXv0.1 models achieve the highest scores, establishing new state‑of‑the‑art performance. performance
Source sentence
Our KodeXv0.1 models set the state of the art in all benchmarks that we tested.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 10 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (33)
- The Theory of Parsing, Translation and Compiling (1972)
- Judging LLM-as-a-judge with MT-bench and Chatbot Arena (2024)
- OLMo: Accelerating the Science of Language Models (2024) — link
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023) — link
- The Llama 3 Herd of Models (2024) — link
- Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2 (2023) — link
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) — link
- PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation (2024) — link
and 25 more
- Experiment Tracking with Weights and Biases (2020) — link
- TRL: Transformer Reinforcement Learning (2020)
- PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods (2022)
- Llm. int8 (): 8-bit matrix multiplication for transformers at scale (2022) — link
- LoRA: Low-Rank Adaptation of Large Language Models (2021) — link
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2023) — link
- Empowering financial futures: Large language models in the modern financial landscape (2024)
- How Multilingual is Multilingual LLM? (2023) — link
- FinanceBench: A New Benchmark for Financial Question Answering (2023) — link
- Training language models to follow instructions with human feedback (2022) — link
- Mistral 7B (2023) — link
- Qwen2 Technical Report (2024) — link
- Stanford Alpaca: An Instruction-following LLaMA model (2023)
- Self-Instruct: Aligning Language Models with Self-Generated Instructions (2023) — link
- Instruction Tuning With Loss Over Instructions (2024) — link
- BloombergGPT: A Large Language Model for Finance (2023) — link
- ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering (2022) — link
- Lighthouzai/finqabench (2024) — link
- FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models (2024) — link
- Instruction Fine-Tuning: Does Prompt Loss Matter? (2024) — link
- Language Models are Unsupervised Multitask Learners (2019)
- Language Models are Few-Shot Learners (2020) — link
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2024) — link
- ORPO: Monolithic Preference Optimization without Reference Model (2024) — link
- Nemotron-4 340B Technical Report (2024) — link
0 of 33 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.