Paper: arXiv 2408.11878

Authors: Jimin Huang, Mengxi Xiao, Dong Li, Zihao Jiang, Yuzhe Yang, Yifei Zhang, Lingfei Qian, Yan Wang, Xueqing Peng, Yang Ren, Ruoyu Xiang, Zhengyu Chen, Xiao Zhang, Yueru He, Weiguang Han, Shunian Chen, Lihang Shen, Daniel Kim, Yangyang Yu, Yupeng Cao, Zhiyang Deng, Haohang Li, Duanyu Feng, Yongfu Dai, VijayaSai Somasundaram, Peng Lu, Guojun Xiong, Zhiwei Liu, Zheheng Luo, Zhiyuan Yao, Ruey-Ling Weng, Meikang Qiu, Kaleb E Smith, Honghai Yu, Yanzhao Lai, Min Peng, Jian-Yun Nie, Jordan W. Suchow, Xiao-Yang Liu, Benyou Wang, Alejandro Lopez-Lira, Qianqian Xie, Sophia Ananiadou, Junichi Tsujii

Abstract

Financial LLMs hold promise for advancing financial tasks and domain-specific applications. However, they are limited by scarce corpora, weak multimodal capabilities, and narrow evaluations, making them less suited for real-world application. To address this, we introduce \textit{“Open-FinLLMs”}, the first open-source multimodal financial LLMs designed to handle diverse tasks across text, tabular, time-series, and chart data, excelling in zero-shot, few-shot, and fine-tuning settings. The suite includes FinLLaMA, pre-trained on a comprehensive 52-billion-token corpus; FinLLaMA-Instruct, fine-tuned with 573K financial instructions; and FinLLaVA, enhanced with 1.43M multimodal tuning pairs for strong cross-modal reasoning. We comprehensively evaluate Open-FinLLMs across 14 financial tasks, 30 datasets, and 4 multimodal tasks in zero-shot, few-shot, and supervised fine-tuning settings, introducing two new multimodal evaluation datasets. Our results show that Open-FinLLMs outperforms afvanced financial and general LLMs such as GPT-4, across financial NLP, decision-making, and multi-modal tasks, highlighting their potential to tackle real-world challenges. To foster innovation and collaboration across academia and industry, we release all codes (https://anonymous.4open.science/r/PIXIU2-0D70/B1D7/LICENSE) and models under OSI-approved licenses.

Complexity vs Empirical Score

  • Math Complexity: 4.0/10
  • Empirical Rigor: 8.5/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper focuses on engineering a suite of open-source LLMs, quantifying performance across extensive tasks and datasets with a lack of advanced mathematical derivations. Its strength lies in heavy implementation, large-scale data curation, and comprehensive benchmarking against real-world models like GPT-4.

Research Flowchart

  flowchart TD
  Goal["Research Goal<br>Develop open-source, multimodal LLMs for financial tasks"]
  Data["Data & Corpora<br>• 52B-token text corpus<br>• 573K financial instructions<br>• 1.43M multimodal image-text pairs"]
  Pretrain["Methodology: Pre-training<br>FinLLaMA model on corpus"]
  Instruct["Methodology: Instruction Tuning<br>FinLLaMA-Instruct on instructions"]
  Multi["Methodology: Multimodal Tuning<br>FinLLaVA on image-text pairs"]
  Eval["Evaluation<br>14 tasks, 30 datasets, 4 multimodal tasks<br>(Zero/Few/Supervised Shot)"]
  Outcome["Key Findings<br>Outperforms GPT-4 & financial LLMs<br>Releases: Models & Code (OSI)"]

  Goal --> Data
  Data --> Pretrain
  Data --> Instruct
  Data --> Multi
  Pretrain --> Instruct
  Instruct --> Multi
  Multi --> Eval
  Eval --> Outcome

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes Open-FinLLMs, an open-source suite of multimodal financial large language models covering text, tabular, time-series, and chart data.

Source sentence
Open-FinLLMs, the first open-source multimodal financial LLMs designed to handle diverse tasks across text, tabular, time-series, and chart data

Assumes

No explicit assumptions found in the text.

Data

Backtest sample: 2021-08 – 2023-04 · single-asset trading dataset (TSLA, COIN, GOOG, NIO) · multi-asset · daily

  • Yahoo Finance data · Yahoo Finance market-prices public
    Source sentence
    The data is sourced from Yahoo Finance, offering a robust foundation for time-series analysis and predictive modeling.
  • SEC Filings · U.S. Securities and Exchange Commission (EDGAR) fundamentals public
    1994-2020
    Source sentence
    SEC Filings: This section includes 6 billion tokens from U.S. SEC filings, such as 10-K and 10-Q reports, spanning from 1994 to 2020.

Recognised sources: SEC EDGAR FiQA Yahoo Finance

Costs

Not modeled A trading or execution result is reported without costs.

Finds 2

Reported performance (FinLLaMA): Sharpe 1.41 (benchmark -0.38) · max drawdown 26.9%

  • FinLLaMA achieves an NER F1 score of 82.10, far surpassing LLaMA3-8B (39.18) and BloombergGPT (60.82). performance
    Source sentence
    In the NER task, FinLLaMA achieves a remarkable F1 score of 82.10, significantly surpassing its backbone model LLaMA3-8B (39.18), LLaMA3.1-8B (49.04), and BloombergGPT (60.82).
  • FinLLaMA attains 85.54 accuracy on the Headlines classification dataset, outperforming LLaMA3-8B (80.59) and BloombergGPT (82.20). performance
    Source sentence
    In the Headlines dataset for classification, FinLLaMA achieves a score of 85.54, outperforming LLaMA3-8B (80.59), LLaMA3.1-8B (59.95), and BloombergGPT (82.20).

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 18 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 10 papers in the archive (5 by independent authors): Interpretable LLMs for Credit Risk: A Systematic Review and … (uses its method or data)

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (206)

  1. Gpt-4 technical report (2023) — link
  2. Fino1: On the Transferability of Reasoning Enhanced LLMs to Finance (2025) — link
  3. Doremi: Optimizing data mixtures speeds up language model pretraining (2024)
  4. MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023) — link
  5. Data-Juicer: A One-Stop Data Processing System for Large Language Models (2024)
  6. FinBen: A Holistic Financial Benchmark for Large Language Models (2024) — link
  7. The FineWeb datasets: Decanting the Web for the finest text data at scale (2024) — link
  8. Llama 2: Open foundation and fine-tuned Chat models (2023) — link
and 198 more
  1. Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks (2023) — link
  2. Large Language Models in Finance: A Survey — → 2311.10723
  3. The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges — → 2304.05351
  4. BloombergGPT : A large language model for finance (2023) — link
  5. Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters (2023)
  6. Stanford Alpaca: An Instruction-following LLaMA model (2023)
  7. FinBERT : Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
  8. A monthly effect in stock returns (1987)
  9. Domain adaption of named entity recognition to support credit risk assessment (2015)
  10. The falcon series of open language models (2023) — link
  11. BELLE: Be Everyone’s Large Language model Engine (2023)
  12. Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases (2023) — link
  13. FinTral: A family of GPT-4 level multimodal financial large language models (2024) — link
  14. Language models are few-shot learners (2020)
  15. Sparks of artificial general intelligence: Early experiments with GPT-4 (2023) — link
  16. Baichuan 2: Open Large-scale Language Models (2023) — link
  17. Overview of the NTCIR-17 FinArg-1 Task: Fine-grained argument understanding in financial analysis (2023)
  18. Training verifiers to solve math word problems (2021) — link
  19. Electra: Pre-training text encoders as discriminators rather than generators (2020) — link
  20. Semeval-2017 task 5: Fine-grained sentiment analysis on financial microblogs and news (2017)
  21. FinQA: A Dataset of Numerical Reasoning over Financial Data (2021)
  22. Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning (2023) — link
  23. Multi-Lingual ESG Issue Identification (2023)
  24. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation (2020)
  25. Tracking the fin trade: genetic stock identification in western Atlantic scalloped hammerhead sharks Sphyrna lewini (2009)
  26. DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning (2023) — link
  27. Data-Centric Financial Large Language Models (2023) — link
  28. Improved baselines with momentum contrastive learning (2020) — link
  29. Lifelong language pretraining with distribution-specialized experts (2023)
  30. S em E val-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News (2017) — link
  31. FinQA: A Dataset of Numerical Reasoning over Financial Data (2022) — link
  32. ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering (2022) — link
  33. MAEC: A Multimodal Aligned Earnings Conference Call Dataset for Financial Risk Prediction (2020) — link
  34. QLoRA: Efficient Finetuning of Quantized LLMs (2023) — link
  35. CALM-Bench: A Multi-task Benchmark for Evaluating Causality-Aware Language Models (2023)
  36. Complementarity, F -score, and NLP Evaluation (2016) — link
  37. LAiW: A Chinese Legal Large Language Models Benchmark (2024) — link
  38. GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2022)
  39. Towards robust and efficient continual language learning (2023) — link
  40. Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models (2024) — link
  41. Enhancing stock movement prediction with adversarial training (2018) — link
  42. GPT-NeoX-20B : An Open-Source Autoregressive Language Model (2022) — link
  43. A probabilistic interpretation of precision, recall and F-score, with implication for evaluation (2005)
  44. GPT-InvestAR: Enhancing Stock Investment Strategies through Annual Report Analysis with Large Language Models — → 2309.03079
  45. Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing (2023) — link
  46. Continual pre-Training of large language models: How to (re)warm your model? (2023)
  47. Multilingual and Cross-Lingual Intent Detection from Spoken Data (2021) — link
  48. Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning — → 2301.10724
  49. Mastering Pair Trading with Risk-Aware Recurrent Reinforcement Learning — → 2304.00364
  50. ECONET : Effective continual pretraining of language models for event temporal reasoning (2021)
  51. How Well Does Self-Supervised Pre-Training Perform with Streaming Data? (2022) — link
  52. Measuring massive multitask language understanding (2020) — link
  53. Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents (2023) — link
  54. F in BART : A Pre-trained Seq2seq Language Model for C hinese Financial Tasks (2023) — link
  55. FinanceBench: A New Benchmark for Financial Question Answering (2023) — link
  56. Mixtral of experts (2024) — link
  57. Mistral 7B (2023) — link
  58. Lifelong pretraining: Continually adapting language models to emerging corpora (2022) — link
  59. Towards continual knowledge learning of language models (2022)
  60. M ulti F in: A Dataset for Multilingual Financial NLP (2023) — link
  61. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019)
  62. Bizbench: A quantitative reasoning benchmark for business and finance (2023) — link
  63. How are We Detecting Inconsistent Method Names? An Empirical Study from Code Review Perspective (2023) — link
  64. Solving quantitative reasoning problems with language models (2022) — link
  65. Decoupled weight decay regularization (2017) — link
  66. The flan collection: Designing data and methods for effective instruction tuning (2023) — link
  67. Textual analogy parsing: What’s shared and what’s compared among analogous facts (2018) — link
  68. Rouge: A package for automatic evaluation of summaries (2004)
  69. Dynamic Datasets and Market Environments for Financial Reinforcement Learning (2023) — link
  70. FinRL-Meta: Market Environments and Benchmarks for Data-Driven Financial Reinforcement Learning (2022) — link
  71. CFGPT: Chinese Financial Assistant with Large Language Model (2023) — link
  72. LVIS: A Dataset for Large Vocabulary Instance Segmentation (2019) — link
  73. Learning Transferable Visual Models From Natural Language Supervision (2021) — link
  74. To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning (2023) — link
  75. The Llama 3 Herd of Models (2024) — link
  76. Visual instruction tuning (2024)
  77. Improved baselines with visual instruction tuning (2024)
  78. TimeLMs : Diachronic language models from Twitter (2022)
  79. Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
  80. Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? An Examination on Several Typical Tasks (2023) — link
  81. Holistic evaluation of language models (2022) — link
  82. BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark (2023) — link
  83. CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes (2023) — link
  84. CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model (2023) — link
  85. FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining (2020) — link
  86. Future climate change significantly alters interannual wheat yield variability over half of harvested areas (2021) — link
  87. A Survey of Large Language Models in Finance (FinLLMs) (2024) — link
  88. F i NER : Financial Numeric Entity Recognition for XBRL Tagging (2022) — link
  89. StockEmotions: Discover Investor Emotions for Financial Sentiment Analysis and Multivariate Time Series — → 2301.09279
  90. Data-centric FinGPT : Democratizing internet-scale data for financial large language models (2023)
  91. FinGPT-HPC : Efficient pretraining and finetuning large language models for financial applications with high-performance computing (2024) — link
  92. FinGPT : Open-source financial large language models — → 2306.06031
  93. Www'18 open challenge: financial opinion mining and question answering (2018)
  94. Good debt or bad debt: Detecting semantic orientations in economic texts (2014)
  95. Financial document causality detection shared task (fincausal 2020) (2020) — link
  96. Ectsum: A new benchmark dataset for bullet point summarization of long earnings call transcripts (2022) — link
  97. Maximum drawdown (2004)
  98. Statlog (German Credit Data) (1994)
  99. Statlog (Australian Credit Approval)
  100. MONOPOLY: Financial Prediction from MONetary POLicY Conference Videos Using Multimodal Cues (2022) — link
  101. C hart QA : A benchmark for question answering about charts with visual and logical reasoning (2022)
  102. TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domains (2024) — link
  103. UniChart: A universal vision-language pretrained model for chart comprehension and reasoning (2023)
  104. Tableformer: Table structure understanding with transformers (2022)
  105. FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis (2023) — link
  106. ALLaVA : Harnessing GPT4V -synthesized data for a lite vision-language model (2024) — link
  107. CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research (2009)
  108. Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts (2013) — link
  109. WWW'18 Open Challenge: Financial Opinion Mining and Question Answering (2018) — link
  110. The Financial Document Causality Detection Shared Task ( F in C ausal 2020) (2020) — link
  111. GPT-4 Technical Report (2023) — link
  112. Training language models to follow instructions with human feedback (2022)
  113. Social Simulacra: Creating Populated Prototypes for Social Computing Systems (2022) — link
  114. Strategic management decision-making in a complex world: quantifying, understanding, and using trade-offs (2017)
  115. Qwen Technical Report (2023) — link
  116. GPT-FinRE: In-context Learning for Financial Relation Extraction using Large Language Models (2023) — link
  117. Improving Language Understanding by Generative Pre-Training (2018) — link
  118. Fin-fact: A benchmark dataset for multimodal financial fact checking and explanation generation (2023) — link
  119. Accurate Stock Movement Prediction with Self-supervised Learning from Sparse Noisy Tweets (2022)
  120. Impact of news on the commodity market: Dataset and results (2021)
  121. Bloom: A 176b-parameter open-access multilingual language model (2022) — link
  122. Multitask Prompted Training Enables Zero-Shot Task Generalization (2022)
  123. Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis — → 2305.07972
  124. Fine-Grained Argument Understanding with BERT Ensemble Techniques: A Deep Dive into Financial Sentiment Analysis (2023)
  125. Finer: Financial named entity recognition dataset and weak-supervision model (2023) — link
  126. FinRED: A dataset for relation extraction in financial domain (2022)
  127. Financial Numeric Extreme Labelling: A dataset and benchmarking (2023)
  128. The sharpe ratio (1998)
  129. Towards reducing hallucination in extracting information from financial reports using Large Language Models — → 2310.10760
  130. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models (2023)
  131. When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain (2022)
  132. Cognitive Architectures for Language Agents (2023) — link
  133. An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels (2022) — link
  134. The Cattell-Horn-Carroll model of intelligence. (2012)
  135. Domain Adaption of Named Entity Recognition to Support Credit Risk Assessment (2015) — link
  136. Gemini: a family of highly capable multimodal models (2023) — link
  137. LLaMA: Open and efficient foundation language models (2023) — link
  138. Fin-Eva Version 1.0 (2023)
  139. Internlm: A multilingual language model with progressively enhanced capabilities (2023)
  140. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (2023) — link
  141. Self-Instruct: Aligning Language Model with Self Generated Instructions (2022) — link
  142. Finetuned language models are zero-shot learners (2021) — link
  143. Hybrid deep sequential modeling for social text-driven stock prediction (2018)
  144. BloombergGPT: A Large Language Model for Finance (2023) — link
  145. FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis (2023) — link
  146. A Comprehensive Survey of Continual Learning: Theory, Method and Application (2024) — link
  147. Larger language models do in-context learning differently (2023) — link
  148. Transformers: State-of-the-Art Natural Language Processing (2020) — link
  149. A Survey on Large Language Model based Autonomous Agents (2023) — link
  150. RecAgent: A Novel Simulation Paradigm for Recommender Systems (2023) — link
  151. FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets — → 2310.04793
  152. MMMU : A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI (2024)
  153. Chart-to-Text: Generating natural language descriptions for charts by adapting the transformer model (2020)
  154. Stock movement prediction from tweets and historical prices (2018)
  155. PIXIU: A large language model, instruction data and evaluation benchmark for finance (2023)
  156. The Wall Street Neophyte: A zero-shot analysis of ChatGPT over multimodal stock movement prediction challenges — → 2304.05351
  157. Accurate multivariate stock movement prediction via data-axis transformer with multi-level contexts (2021)
  158. Bartscore: Evaluating generated text as text generation (2021)
  159. Back to the Future: Towards Explainable Temporal Reasoning with Large Language Models (2023) — link
  160. InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning — → 2309.13064
  161. FinMem : A performance-enhanced LLM trading agent with layered memory and character design — → 2311.13743
  162. Generating plausible counterfactual explanations for deep transformers in financial text classification (2020) — link
  163. Docmath-eval: Evaluating numerical reasoning capabilities of llms in understanding long documents with tabular data (2023) — link
  164. KnowledgeMath: Knowledge-Intensive Math Word Problem Solving in Finance Domains (2023) — link
  165. Opt: Open pre-trained transformer language models (2022) — link
  166. Mengzi: Towards lightweight yet ingenious pre-trained models for chinese (2021) — link
  167. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance (2021) — link
  168. Trade the event: Corporate events detection for news-based event-driven trading (2021) — link
  169. Bertscore: Evaluating text generation with bert (2019) — link
  170. Forecasting the equity premium: Do deep neural network models work? (2023)
  171. D'olares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English (2024) — link
  172. Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models — → 2310.04027
  173. Continual Learning with Pre-Trained Models: A Survey (2024) — link
  174. Instruction Tuning for Large Language Models: A Survey (2023) — link
  175. CGCE: A Chinese Generative Chat Evaluation Benchmark for General and Financial Domains (2023) — link
  176. Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context (2021)
  177. Measuring Massive Multitask Language Understanding (2021)
  178. Aligning AI With Shared Human Values (2021)
  179. Instruct-fingpt: Financial sentiment analysis by instruction tuning of general-purpose large language models — → 2306.12659
  180. GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023) — link
  181. Evaluating Large Language Models Trained on Code (2021) — link
  182. Training Verifiers to Solve Math Word Problems (2021) — link
  183. Measuring Mathematical Problem Solving With the MATH Dataset (2021) — link
  184. A survey of large language models for financial applications: Progress, Prospects and Challenges — → 2406.11903
  185. Pre-trained language models in biomedical domain: A systematic survey (2023)
  186. Continual learning for large language models: A survey (2024) — link
  187. A study of generative large language model for medical research and healthcare (2023)
  188. Code llama: Open foundation models for code (2023) — link
  189. PMC-LLaMA: toward building open-source language models for medicine (2024)
  190. Continual Pre-Training of Large Language Models: How to re-warm your model?
  191. Improving language understanding by generative pre-training
  192. Qlora: Efficient finetuning of quantized llms (2024)
  193. Decoupled Weight Decay Regularization
  194. The finben: An holistic financial benchmark for large language models (2024) — link
  195. Risk Management and Financial Institutions (2007)
  196. The Sharpe Ratio (1994)
  197. Downside Risk (2003)
  198. Volatility Tests and Efficient Markets: A Review Essay (1988)

17 of 206 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.