Paper: arXiv 2408.11878
Authors: Jimin Huang, Mengxi Xiao, Dong Li, Zihao Jiang, Yuzhe Yang, Yifei Zhang, Lingfei Qian, Yan Wang, Xueqing Peng, Yang Ren, Ruoyu Xiang, Zhengyu Chen, Xiao Zhang, Yueru He, Weiguang Han, Shunian Chen, Lihang Shen, Daniel Kim, Yangyang Yu, Yupeng Cao, Zhiyang Deng, Haohang Li, Duanyu Feng, Yongfu Dai, VijayaSai Somasundaram, Peng Lu, Guojun Xiong, Zhiwei Liu, Zheheng Luo, Zhiyuan Yao, Ruey-Ling Weng, Meikang Qiu, Kaleb E Smith, Honghai Yu, Yanzhao Lai, Min Peng, Jian-Yun Nie, Jordan W. Suchow, Xiao-Yang Liu, Benyou Wang, Alejandro Lopez-Lira, Qianqian Xie, Sophia Ananiadou, Junichi Tsujii
Abstract
Financial LLMs hold promise for advancing financial tasks and domain-specific applications. However, they are limited by scarce corpora, weak multimodal capabilities, and narrow evaluations, making them less suited for real-world application. To address this, we introduce \textit{“Open-FinLLMs”}, the first open-source multimodal financial LLMs designed to handle diverse tasks across text, tabular, time-series, and chart data, excelling in zero-shot, few-shot, and fine-tuning settings. The suite includes FinLLaMA, pre-trained on a comprehensive 52-billion-token corpus; FinLLaMA-Instruct, fine-tuned with 573K financial instructions; and FinLLaVA, enhanced with 1.43M multimodal tuning pairs for strong cross-modal reasoning. We comprehensively evaluate Open-FinLLMs across 14 financial tasks, 30 datasets, and 4 multimodal tasks in zero-shot, few-shot, and supervised fine-tuning settings, introducing two new multimodal evaluation datasets. Our results show that Open-FinLLMs outperforms afvanced financial and general LLMs such as GPT-4, across financial NLP, decision-making, and multi-modal tasks, highlighting their potential to tackle real-world challenges. To foster innovation and collaboration across academia and industry, we release all codes (https://anonymous.4open.science/r/PIXIU2-0D70/B1D7/LICENSE) and models under OSI-approved licenses.
Complexity vs Empirical Score
- Math Complexity: 4.0/10
- Empirical Rigor: 8.5/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper focuses on engineering a suite of open-source LLMs, quantifying performance across extensive tasks and datasets with a lack of advanced mathematical derivations. Its strength lies in heavy implementation, large-scale data curation, and comprehensive benchmarking against real-world models like GPT-4.
Research Flowchart
flowchart TD Goal["Research Goal<br>Develop open-source, multimodal LLMs for financial tasks"] Data["Data & Corpora<br>• 52B-token text corpus<br>• 573K financial instructions<br>• 1.43M multimodal image-text pairs"] Pretrain["Methodology: Pre-training<br>FinLLaMA model on corpus"] Instruct["Methodology: Instruction Tuning<br>FinLLaMA-Instruct on instructions"] Multi["Methodology: Multimodal Tuning<br>FinLLaVA on image-text pairs"] Eval["Evaluation<br>14 tasks, 30 datasets, 4 multimodal tasks<br>(Zero/Few/Supervised Shot)"] Outcome["Key Findings<br>Outperforms GPT-4 & financial LLMs<br>Releases: Models & Code (OSI)"] Goal --> Data Data --> Pretrain Data --> Instruct Data --> Multi Pretrain --> Instruct Instruct --> Multi Multi --> Eval Eval --> Outcome
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes Open-FinLLMs, an open-source suite of multimodal financial large language models covering text, tabular, time-series, and chart data.
Source sentence
Open-FinLLMs, the first open-source multimodal financial LLMs designed to handle diverse tasks across text, tabular, time-series, and chart data
Assumes
No explicit assumptions found in the text.
Data
2021-08 – 2023-04 · single-asset trading dataset (TSLA, COIN, GOOG, NIO) · multi-asset · daily
- Yahoo Finance data market-prices public
Source sentence
The data is sourced from Yahoo Finance, offering a robust foundation for time-series analysis and predictive modeling.
- SEC Filings fundamentals public
Source sentence
SEC Filings: This section includes 6 billion tokens from U.S. SEC filings, such as 10-K and 10-Q reports, spanning from 1994 to 2020.
SEC EDGAR FiQA Yahoo Finance
Costs
Not modeled
Finds 2
Sharpe 1.41 (benchmark -0.38) · max drawdown 26.9%
- FinLLaMA achieves an NER F1 score of 82.10, far surpassing LLaMA3-8B (39.18) and BloombergGPT (60.82). performance
Source sentence
In the NER task, FinLLaMA achieves a remarkable F1 score of 82.10, significantly surpassing its backbone model LLaMA3-8B (39.18), LLaMA3.1-8B (49.04), and BloombergGPT (60.82).
- FinLLaMA attains 85.54 accuracy on the Headlines classification dataset, outperforming LLaMA3-8B (80.59) and BloombergGPT (82.20). performance
Source sentence
In the Headlines dataset for classification, FinLLaMA achieves a score of 85.54, outperforming LLaMA3-8B (80.59), LLaMA3.1-8B (59.95), and BloombergGPT (82.20).
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 18 related papers). Coverage is partial, so this is neither replication nor refutation.
10 papers in the archive : Interpretable LLMs for Credit Risk: A Systematic Review and …
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (206)
- Gpt-4 technical report (2023) — link
- Fino1: On the Transferability of Reasoning Enhanced LLMs to Finance (2025) — link
- Doremi: Optimizing data mixtures speeds up language model pretraining (2024)
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023) — link
- Data-Juicer: A One-Stop Data Processing System for Large Language Models (2024)
- FinBen: A Holistic Financial Benchmark for Large Language Models (2024) — link
- The FineWeb datasets: Decanting the Web for the finest text data at scale (2024) — link
- Llama 2: Open foundation and fine-tuned Chat models (2023) — link
and 198 more
- Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks (2023) — link
- Large Language Models in Finance: A Survey — → 2311.10723
- The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges — → 2304.05351
- BloombergGPT : A large language model for finance (2023) — link
- Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters (2023)
- Stanford Alpaca: An Instruction-following LLaMA model (2023)
- FinBERT : Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
- A monthly effect in stock returns (1987)
- Domain adaption of named entity recognition to support credit risk assessment (2015)
- The falcon series of open language models (2023) — link
- BELLE: Be Everyone’s Large Language model Engine (2023)
- Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases (2023) — link
- FinTral: A family of GPT-4 level multimodal financial large language models (2024) — link
- Language models are few-shot learners (2020)
- Sparks of artificial general intelligence: Early experiments with GPT-4 (2023) — link
- Baichuan 2: Open Large-scale Language Models (2023) — link
- Overview of the NTCIR-17 FinArg-1 Task: Fine-grained argument understanding in financial analysis (2023)
- Training verifiers to solve math word problems (2021) — link
- Electra: Pre-training text encoders as discriminators rather than generators (2020) — link
- Semeval-2017 task 5: Fine-grained sentiment analysis on financial microblogs and news (2017)
- FinQA: A Dataset of Numerical Reasoning over Financial Data (2021)
- Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning (2023) — link
- Multi-Lingual ESG Issue Identification (2023)
- The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation (2020)
- Tracking the fin trade: genetic stock identification in western Atlantic scalloped hammerhead sharks Sphyrna lewini (2009)
- DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning (2023) — link
- Data-Centric Financial Large Language Models (2023) — link
- Improved baselines with momentum contrastive learning (2020) — link
- Lifelong language pretraining with distribution-specialized experts (2023)
- S em E val-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News (2017) — link
- FinQA: A Dataset of Numerical Reasoning over Financial Data (2022) — link
- ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering (2022) — link
- MAEC: A Multimodal Aligned Earnings Conference Call Dataset for Financial Risk Prediction (2020) — link
- QLoRA: Efficient Finetuning of Quantized LLMs (2023) — link
- CALM-Bench: A Multi-task Benchmark for Evaluating Causality-Aware Language Models (2023)
- Complementarity, F -score, and NLP Evaluation (2016) — link
- LAiW: A Chinese Legal Large Language Models Benchmark (2024) — link
- GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2022)
- Towards robust and efficient continual language learning (2023) — link
- Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models (2024) — link
- Enhancing stock movement prediction with adversarial training (2018) — link
- GPT-NeoX-20B : An Open-Source Autoregressive Language Model (2022) — link
- A probabilistic interpretation of precision, recall and F-score, with implication for evaluation (2005)
- GPT-InvestAR: Enhancing Stock Investment Strategies through Annual Report Analysis with Large Language Models — → 2309.03079
- Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing (2023) — link
- Continual pre-Training of large language models: How to (re)warm your model? (2023)
- Multilingual and Cross-Lingual Intent Detection from Spoken Data (2021) — link
- Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning — → 2301.10724
- Mastering Pair Trading with Risk-Aware Recurrent Reinforcement Learning — → 2304.00364
- ECONET : Effective continual pretraining of language models for event temporal reasoning (2021)
- How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2022) — link
- Measuring massive multitask language understanding (2020) — link
- Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents (2023) — link
- F in BART : A Pre-trained Seq2seq Language Model for C hinese Financial Tasks (2023) — link
- FinanceBench: A New Benchmark for Financial Question Answering (2023) — link
- Mixtral of experts (2024) — link
- Mistral 7B (2023) — link
- Lifelong pretraining: Continually adapting language models to emerging corpora (2022) — link
- Towards continual knowledge learning of language models (2022)
- M ulti F in: A Dataset for Multilingual Financial NLP (2023) — link
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019)
- Bizbench: A quantitative reasoning benchmark for business and finance (2023) — link
- How are We Detecting Inconsistent Method Names? An Empirical Study from Code Review Perspective (2023) — link
- Solving quantitative reasoning problems with language models (2022) — link
- Decoupled weight decay regularization (2017) — link
- The flan collection: Designing data and methods for effective instruction tuning (2023) — link
- Textual analogy parsing: What’s shared and what’s compared among analogous facts (2018) — link
- Rouge: A package for automatic evaluation of summaries (2004)
- Dynamic Datasets and Market Environments for Financial Reinforcement Learning (2023) — link
- FinRL-Meta: Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning (2022) — link
- CFGPT: Chinese Financial Assistant with Large Language Model (2023) — link
- LVIS: A Dataset for Large Vocabulary Instance Segmentation (2019) — link
- Learning Transferable Visual Models From Natural Language Supervision (2021) — link
- To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning (2023) — link
- The Llama 3 Herd of Models (2024) — link
- Visual instruction tuning (2024)
- Improved baselines with visual instruction tuning (2024)
- TimeLMs : Diachronic language models from Twitter (2022)
- Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
- Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? An Examination on Several Typical Tasks (2023) — link
- Holistic evaluation of language models (2022) — link
- BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark (2023) — link
- CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes (2023) — link
- CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model (2023) — link
- FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining (2020) — link
- Future climate change significantly alters interannual wheat yield variability over half of harvested areas (2021) — link
- A Survey of Large Language Models in Finance (FinLLMs) (2024) — link
- F i NER : Financial Numeric Entity Recognition for XBRL Tagging (2022) — link
- StockEmotions: Discover Investor Emotions for Financial Sentiment Analysis and Multivariate Time Series — → 2301.09279
- Data-centric FinGPT : Democratizing internet-scale data for financial large language models (2023)
- FinGPT-HPC : Efficient pretraining and finetuning large language models for financial applications with high-performance computing (2024) — link
- FinGPT : Open-source financial large language models — → 2306.06031
- Www'18 open challenge: financial opinion mining and question answering (2018)
- Good debt or bad debt: Detecting semantic orientations in economic texts (2014)
- Financial document causality detection shared task (fincausal 2020) (2020) — link
- Ectsum: A new benchmark dataset for bullet point summarization of long earnings call transcripts (2022) — link
- Maximum drawdown (2004)
- Statlog (German Credit Data) (1994)
- Statlog (Australian Credit Approval)
- MONOPOLY: Financial Prediction from MONetary POLicY Conference Videos Using Multimodal Cues (2022) — link
- C hart QA : A benchmark for question answering about charts with visual and logical reasoning (2022)
- TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domains (2024) — link
- UniChart: A universal vision-language pretrained model for chart comprehension and reasoning (2023)
- Tableformer: Table structure understanding with transformers (2022)
- FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis (2023) — link
- ALLaVA : Harnessing GPT4V -synthesized data for a lite vision-language model (2024) — link
- CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research (2009)
- Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts (2013) — link
- WWW'18 Open Challenge: Financial Opinion Mining and Question Answering (2018) — link
- The Financial Document Causality Detection Shared Task ( F in C ausal 2020) (2020) — link
- GPT-4 Technical Report (2023) — link
- Training language models to follow instructions with human feedback (2022)
- Social Simulacra: Creating Populated Prototypes for Social Computing Systems (2022) — link
- Strategic management decision-making in a complex world: quantifying, understanding, and using trade-offs (2017)
- Qwen Technical Report (2023) — link
- GPT-FinRE: In-context Learning for Financial Relation Extraction using Large Language Models (2023) — link
- Improving Language Understanding by Generative Pre-Training (2018) — link
- Fin-fact: A benchmark dataset for multimodal financial fact checking and explanation generation (2023) — link
- Accurate Stock Movement Prediction with Self-supervised Learning from Sparse Noisy Tweets (2022)
- Impact of news on the commodity market: Dataset and results (2021)
- Bloom: A 176b-parameter open-access multilingual language model (2022) — link
- Multitask Prompted Training Enables Zero-Shot Task Generalization (2022)
- Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis — → 2305.07972
- Fine-Grained Argument Understanding with BERT Ensemble Techniques: A Deep Dive into Financial Sentiment Analysis (2023)
- Finer: Financial named entity recognition dataset and weak-supervision model (2023) — link
- FinRED: A dataset for relation extraction in financial domain (2022)
- Financial Numeric Extreme Labelling: A dataset and benchmarking (2023)
- The sharpe ratio (1998)
- Towards reducing hallucination in extracting information from financial reports using Large Language Models — → 2310.10760
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models (2023)
- When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain (2022)
- Cognitive Architectures for Language Agents (2023) — link
- An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels (2022) — link
- The Cattell-Horn-Carroll model of intelligence. (2012)
- Domain Adaption of Named Entity Recognition to Support Credit Risk Assessment (2015) — link
- Gemini: a family of highly capable multimodal models (2023) — link
- LLaMA: Open and efficient foundation language models (2023) — link
- Fin-Eva Version 1.0 (2023)
- Internlm: A multilingual language model with progressively enhanced capabilities (2023)
- Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (2023) — link
- Self-Instruct: Aligning Language Model with Self Generated Instructions (2022) — link
- Finetuned language models are zero-shot learners (2021) — link
- Hybrid deep sequential modeling for social text-driven stock prediction (2018)
- BloombergGPT: A Large Language Model for Finance (2023) — link
- FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis (2023) — link
- A Comprehensive Survey of Continual Learning: Theory, Method and Application (2024) — link
- Larger language models do in-context learning differently (2023) — link
- Transformers: State-of-the-Art Natural Language Processing (2020) — link
- A Survey on Large Language Model based Autonomous Agents (2023) — link
- RecAgent: A Novel Simulation Paradigm for Recommender Systems (2023) — link
- FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets — → 2310.04793
- MMMU : A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI (2024)
- Chart-to-Text: Generating natural language descriptions for charts by adapting the transformer model (2020)
- Stock movement prediction from tweets and historical prices (2018)
- PIXIU: A large language model, instruction data and evaluation benchmark for finance (2023)
- The Wall Street Neophyte: A zero-shot analysis of ChatGPT over multimodal stock movement prediction challenges — → 2304.05351
- Accurate multivariate stock movement prediction via data-axis transformer with multi-level contexts (2021)
- Bartscore: Evaluating generated text as text generation (2021)
- Back to the Future: Towards Explainable Temporal Reasoning with Large Language Models (2023) — link
- InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning — → 2309.13064
- FinMem : A performance-enhanced LLM trading agent with layered memory and character design — → 2311.13743
- Generating plausible counterfactual explanations for deep transformers in financial text classification (2020) — link
- Docmath-eval: Evaluating numerical reasoning capabilities of llms in understanding long documents with tabular data (2023) — link
- KnowledgeMath: Knowledge-Intensive Math Word Problem Solving in Finance Domains (2023) — link
- Opt: Open pre-trained transformer language models (2022) — link
- Mengzi: Towards lightweight yet ingenious pre-trained models for chinese (2021) — link
- TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance (2021) — link
- Trade the event: Corporate events detection for news-based event-driven trading (2021) — link
- Bertscore: Evaluating text generation with bert (2019) — link
- Forecasting the equity premium: Do deep neural network models work? (2023)
- D'olares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English (2024) — link
- Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models — → 2310.04027
- Continual Learning with Pre-Trained Models: A Survey (2024) — link
- Instruction Tuning for Large Language Models: A Survey (2023) — link
- CGCE: A Chinese Generative Chat Evaluation Benchmark for General and Financial Domains (2023) — link
- Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context (2021)
- Measuring Massive Multitask Language Understanding (2021)
- Aligning AI With Shared Human Values (2021)
- Instruct-fingpt: Financial sentiment analysis by instruction tuning of general-purpose large language models — → 2306.12659
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023) — link
- Evaluating Large Language Models Trained on Code (2021) — link
- Training Verifiers to Solve Math Word Problems (2021) — link
- Measuring Mathematical Problem Solving With the MATH Dataset (2021) — link
- A survey of large language models for financial applications: Progress, Prospects and Challenges — → 2406.11903
- Pre-trained language models in biomedical domain: A systematic survey (2023)
- Continual learning for large language models: A survey (2024) — link
- A study of generative large language model for medical research and healthcare (2023)
- Code llama: Open foundation models for code (2023) — link
- PMC-LLaMA: toward building open-source language models for medicine (2024)
- Continual Pre-Training of Large Language Models: How to re-warm your model?
- Improving language understanding by generative pre-training
- Qlora: Efficient finetuning of quantized llms (2024)
- Decoupled Weight Decay Regularization
- The finben: An holistic financial benchmark for large language models (2024) — link
- Risk Management and Financial Institutions (2007)
- The Sharpe Ratio (1994)
- Downside Risk (2003)
- Volatility Tests and Efficient Markets: A Review Essay (1988)
17 of 206 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.