# Towards Temporal-Aware Multi-Modal Retrieval Augmented Generation in Finance

arXiv: https://arxiv.org/abs/2503.05185 · HTML: https://mlsanity.com/paper/2503.05185/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Placed in

- Retrieval-augmented generation / [Legal and financial RAG](https://mlsanity.com/rag/legal-and-financial-rag/)

## Builds on (most cited first, 16 of 16)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts](https://mlsanity.com/paper/1307.5336/) (2013)
- [FinanceBench: A New Benchmark for Financial Question Answering](https://mlsanity.com/paper/2311.11944/) (2023)
- [MultiModalQA: Complex Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2104.06039/) (2021)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data](https://mlsanity.com/paper/2206.01347/) (2022)
- [HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction](https://mlsanity.com/paper/2408.04948/) (2024)
- [Towards Complex Document Understanding By Discrete Reasoning](https://mlsanity.com/paper/2207.11871/) (2022)
- [VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval](https://mlsanity.com/paper/2406.04292/) (2024)
- [FinTextQA: A Dataset for Long-form Financial Question Answering](https://mlsanity.com/paper/2405.09980/) (2024)
- [AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework](https://mlsanity.com/paper/2403.12582/) (2024)
- [OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain](https://mlsanity.com/paper/2412.13018/) (2024)

## Built on by (most cited first, 8 of 8)

- [Reading Between the Timelines: RAG for Answering Diachronic Questions](https://mlsanity.com/paper/2507.22917/) (2025)
- [FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation](https://mlsanity.com/paper/2602.03130/) (2026)
- [Time is Not a Label: Continuous Phase Rotation for Temporal Knowledge Graphs and Agentic Memory](https://mlsanity.com/paper/2604.11544/) (2026)
- [T2-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.12071/) (2025)
- [Multimedia and Visual Analytics in the Agentic Era](https://mlsanity.com/paper/2504.06138/) (2025)
- [Strat-LLM: Stratified Strategy Alignment for LLM-based Stock Trading with Real-time Multi-Source Signals](https://mlsanity.com/paper/2605.06024/) (2026)
- [History Rhymes: Macro-Contextual Retrieval for Robust Financial Forecasting](https://mlsanity.com/paper/2511.09754/) (2025)
- [Frontier Financial Judgement: Can agents tell what might move a stock?](https://mlsanity.com/paper/2607.20645/) (2026)

## Abstract (arXiv)

Finance decision-making often relies on in-depth data analysis across various data sources, including financial tables, news articles, stock prices, etc. In this work, we introduce FinTMMBench, the first comprehensive benchmark for evaluating temporal-aware multi-modal Retrieval-Augmented Generation (RAG) systems in finance. Built from heterologous data of NASDAQ 100 companies, FinTMMBench offers three significant advantages. 1) Multi-modal Corpus: It encompasses a hybrid of financial tables, news articles, daily stock prices, and visual technical charts as the corpus. 2) Temporal-aware Questions: Each question requires the retrieval and interpretation of its relevant data over a specific time period, including daily, weekly, monthly, quarterly, and annual periods. 3) Diverse Financial Analysis Tasks: The questions involve 10 different financial analysis tasks designed by domain experts, including information extraction, trend analysis, sentiment analysis and event detection, etc. We further propose a novel TMMHybridRAG method, which first leverages LLMs to convert data from other modalities (e.g., tabular, visual and time-series data) into textual format and then incorporates temporal information in each node when constructing graphs and dense indexes. Its effectiveness has been validated in extensive experiments, but notable gaps remain, highlighting the challenges presented by our FinTMMBench.