Paper: arXiv 2503.05185

Authors: Fengbin Zhu, Junfeng Li, Liangming Pan, Wenjie Wang, Fuli Feng, Chao Wang, Huanbo Luan, Tat-Seng Chua

Abstract

Finance decision-making often relies on in-depth data analysis across various data sources, including financial tables, news articles, stock prices, etc. In this work, we introduce FinTMMBench, the first comprehensive benchmark for evaluating temporal-aware multi-modal Retrieval-Augmented Generation (RAG) systems in finance. Built from heterologous data of NASDAQ 100 companies, FinTMMBench offers three significant advantages. 1) Multi-modal Corpus: It encompasses a hybrid of financial tables, news articles, daily stock prices, and visual technical charts as the corpus. 2) Temporal-aware Questions: Each question requires the retrieval and interpretation of its relevant data over a specific time period, including daily, weekly, monthly, quarterly, and annual periods. 3) Diverse Financial Analysis Tasks: The questions involve 10 different financial analysis tasks designed by domain experts, including information extraction, trend analysis, sentiment analysis and event detection, etc. We further propose a novel TMMHybridRAG method, which first leverages LLMs to convert data from other modalities (e.g., tabular, visual and time-series data) into textual format and then incorporates temporal information in each node when constructing graphs and dense indexes. Its effectiveness has been validated in extensive experiments, but notable gaps remain, highlighting the challenges presented by our FinTMMBench.

Complexity vs Empirical Score

  • Math Complexity: 2.5/10
  • Empirical Rigor: 8.5/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper introduces a new benchmark and a novel RAG method, but the methodology relies more on LLM-based data conversion and retrieval techniques rather than advanced mathematical modeling. It demonstrates high empirical rigor through the construction of a comprehensive, multi-modal dataset (FinTMMBench) with specific data sources and evaluation metrics, and proposes a method with implementation details and reported results.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Develop Temporal-Aware<br>Multi-Modal RAG for Finance"] --> B["Construct FinTMMBench Benchmark"]
  B --> C{"Multi-modal Corpus<br>Temporal-aware Questions<br>Diverse Analysis Tasks"}
  C --> D["TMMHybridRAG Methodology"]
  D --> E["Modality Conversion<br>LLMs convert tables/charts<br>to text"]
  E --> F["Temporal Graph Construction<br>Nodes encode time segments"]
  E --> G["Dense Indexing<br>Embed temporal info"]
  F --> H["Retrieval & Generation<br>Query relevant data<br>across time periods"]
  G --> H
  H --> I["Experimental Results"]
  I --> J{"Key Findings/Outcomes"}
  J --> K["Validated TMMHybridRAG Effectiveness"]
  J --> L["Identified Temporal Challenges<br>in Multi-modal Finance"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes TMMHybridRAG, a novel temporal-aware multi-modal Retrieval-Augmented Generation method that integrates dense and graph retrieval techniques, and introduces the FinTMMBench benchmark.

Source sentence
we propose TMMHybridRAG, a novel temporal-aware multi-modal RAG method that integrates dense and graph retrieval techniques.

Assumes

No explicit assumptions found in the text.

Data

  • FinTMMBench · NASDAQ-100 companies benchmark public
    2022 · NASDAQ-100 companies
    Source sentence
    To construct FinTMMBench, we collect financial data of the NASDAQ-100 companies in 2022, which include four types as below.
  • FinTMMBench raw data · NASDAQ-100 companies benchmark public
    NASDAQ-100 companies
    Source sentence
    FinTMMBench consists of 34, 815 raw data entries from NASDAQ-100 companies across four modalities, including 1, 500 financial tables, 3, 133 news articles, 25, 200 daily stock price records, and 6, 267 visual technical charts.

Costs

Not applicable The paper reports no trading or execution result.

Finds 2

  • TMMHybridRAG (All) achieves the best results across all evaluation metrics, with EM 19.12%, F1 31.41%, accuracy 26.56%, and LLM-judge accuracy 21.53%. performance
    Source sentence
    ... it attains an EM score of 19.12%, an F1 score of 31.41, an accuracy of 26.56%, and an LLM-judge accuracy of 21.53%.
  • Despite being state-of-the-art, the F1 score remains relatively low at 31.41, highlighting substantial challenges in FinTMMBench. negative
    Source sentence
    Though our TMMHybridRAG (All) achieves state-of-the-art on FinTMMBench, the F1 score remains relatively low at 31.41. This highlights the significant challenges inherent in FinTMMBench.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 13 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 3 papers in the archive (2 by independent authors): When AI Trading Agents Compete: Adverse Selection of … (uses its method or data)

Code

https://github.com/lijunfeng99/FinTMMBench — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (75)

  1. UK List of \TeX Frequently Asked Questions (2019)
  2. The \textsf amsart , \textsf amsproc , and \textsf amsbook document~classes (2004)
  3. a lgorithm2e.sty—package for algorithms (2015)
  4. The algorithms bundle (2009)
  5. The Listings Package (2015)
  6. Publication quality tables in \LaTeX (2005)
  7. ACM Visual Identity Standards (2007)
  8. The subcaption package (2013)
and 67 more
  1. A package to create a nomenclature (2005)
  2. User Manual for glossaries.sty v4.44 (2019)
  3. The \textsl textcase package (2004)
  4. Babel (2022)
  5. Gpt-4 technical report (2023) — link
  6. Retrieval-augmented generation for knowledge-intensive nlp tasks (2020)
  7. A survey of large language models (2023) — link
  8. M ulti H iertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data (2022)
  9. Retrieving and reading: A comprehensive survey on open-domain question answering (2021) — link
  10. Soargraph: Numerical reasoning over financial table-text data via semantic-oriented hierarchical graphs (2023)
  11. Doc2SoarGraph: Discrete reasoning over visually-rich table-text documents via semantic-oriented hierarchical graphs (2023) — link
  12. VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval (2024) — link
  13. Learning transferable visual models from natural language supervision (2021)
  14. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation (2022)
  15. C-Pack: Packaged Resources To Advance General Chinese Embedding (2023) — link
  16. HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction — → 2408.04948
  17. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) — link
  18. The finben: An holistic financial benchmark for large language models (2024) — link
  19. The llama 3 herd of models (2024) — link
  20. Constitutional ai: Harmlessness from ai feedback (2022) — link
  21. Poor numbers: how we are misled by African development statistics and what to do about it (2013)
  22. Know what you don’t know: Unanswerable questions for SQuAD (2018) — link
  23. MultiModalQA: complex question answering over text, tables and images
  24. Tempquestions: A benchmark for temporal question answering (2018)
  25. Survey on Temporal Knowledge Graph (2021) — link
  26. Check your facts and try again: Improving large language models with external knowledge and automated feedback (2023) — link
  27. Improving retrieval for rag based question answering models on financial documents (2024) — link
  28. Financial report chunking for effective retrieval augmented generation (2024) — link
  29. Enhancing financial sentiment analysis via retrieval augmented large language models — → 2310.04027
  30. Revolutionizing retrieval-augmented generation with enhanced PDF structure recognition (2024) — link
  31. Financebench: A new benchmark for financial question answering (2023) — link
  32. Www'18 open challenge: financial opinion mining and question answering (2018)
  33. AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework (2024) — link
  34. Good debt or bad debt: Detecting semantic orientations in economic texts (2014)
  35. Finqa: A dataset of numerical reasoning over financial data (2021) — link
  36. OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain (2024) — link
  37. FinTextQA: A Dataset for Long-form Financial Question Answering (2024) — link
  38. Security Analysis (1934)
  39. Why Does the Effective Context Length of LLMs Fall Short? (2024) — link
  40. Introducing the Knowledge Graph: Things, Not Strings (2012) — link
  41. Retrieval-Augmented Generation Meets Data-Driven Tabula Rasa Approach for Temporal Knowledge Graph Forecasting (2024) — link
  42. From local to global: A graph rag approach to query-focused summarization (2024) — link
  43. M \textsuperscript 3 CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought (2024) — link
  44. Is chatgpt a good nlg evaluator? a preliminary study (2023) — link
  45. Judging llm-as-a-judge with mt-bench and chatbot arena (2023)
  46. Retrieval-augmented generation for large language models: A survey (2023) — link
  47. Lightrag: Simple and fast retrieval-augmented generation (2024)
  48. Squad: 100,000+ questions for machine comprehension of text (2016) — link
  49. The probabilistic relevance framework: BM25 and beyond (2009)
  50. Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (2023) — link
  51. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025) — link
  52. Qwen2 Technical Report (2024) — link
  53. TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance (2021)
  54. Learning to imagine: Integrating counterfactual thinking in neural discrete reasoning (2022)
  55. Towards complex document understanding by discrete reasoning (2022)
  56. The cross-section of expected stock returns (1992)
  57. Foundations of portfolio theory (1991)
  58. The risk management of everything (2004)
  59. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities (2025) — link
  60. Scilab — link
  61. Scilab — link
  62. The Computational Geometry Algorithms Library — link
  63. The Computational Geometry Algorithms Library — link
  64. The Parmap library — link
  65. The Parmap library — link
  66. BabyJ : from Object Based to Class Based Programming via Types (2003)
  67. tag-sound

2 of 75 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.