# Multi-Document Financial Question Answering using LLMs

arXiv: https://arxiv.org/abs/2411.07264 · HTML: https://mlsanity.com/paper/2411.07264/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://mlsanity.com/paper/2309.15217/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://mlsanity.com/paper/2405.05904/) (2024)
- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [GPT-NER: Named Entity Recognition via Large Language Models](https://mlsanity.com/paper/2304.10428/) (2023)
- [FinBen: A Holistic Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2402.12659/) (2024)
- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction](https://mlsanity.com/paper/2408.04948/) (2024)
- [Searching for Best Practices in Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.01219/) (2024)
- [A Survey of Large Language Models in Finance (FinLLMs)](https://mlsanity.com/paper/2402.02315/) (2024)
- [FinGPT: Democratizing Internet-scale Data for Financial Large Language Models](https://mlsanity.com/paper/2307.10485/) (2023)
- [From LLM to NMT: Advancing Low-Resource Machine Translation with Claude](https://mlsanity.com/paper/2404.13813/) (2024)
- [KG-RAG: Bridging the Gap Between Knowledge and Creativity](https://mlsanity.com/paper/2405.12035/) (2024)
- [From human experts to machines: An LLM supported approach to ontology and knowledge graph construction](https://mlsanity.com/paper/2403.08345/) (2024)
- [Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains](https://mlsanity.com/paper/2402.05140/) (2024)
- [Embedding-based Retrieval with LLM for Effective Agriculture Information Extracting from Unstructured Data](https://mlsanity.com/paper/2308.03107/) (2023)
- [MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks](https://mlsanity.com/paper/2404.00457/) (2024)
- [Does Biomedical Training Lead to Better Medical Performance?](https://mlsanity.com/paper/2404.04067/) (2024)
- [Towards Universal Semantic Tagging](https://mlsanity.com/paper/1709.10381/) (2017)
- [LLMs in Biomedicine: A study on clinical Named Entity Recognition](https://mlsanity.com/paper/2404.07376/) (2024)
- [Investigating Answerability of LLMs for Long-Form Question Answering](https://mlsanity.com/paper/2309.08210/) (2023)
- [WHISMA: A Speech-LLM to Perform Zero-shot Spoken Language Understanding](https://mlsanity.com/paper/2408.16423/) (2024)
- [DEXTER: A Benchmark for open-domain Complex Question Answering using LLMs](https://mlsanity.com/paper/2406.17158/) (2024)

## Built on by (most cited first, 6 of 6)

- [FinSage: A Multi-aspect RAG System for Financial Filings Question Answering](https://mlsanity.com/paper/2504.14493/) (2025)
- [Structuring the Unstructured: A Multi-Agent System for Extracting and Querying Financial KPIs and Guidance](https://mlsanity.com/paper/2505.19197/) (2025)
- [FinReflectKG -- MultiHop: Financial QA Benchmark for Reasoning with Knowledge Graph Evidence](https://mlsanity.com/paper/2510.02906/) (2025)
- [Parallel and Multi-Stage Knowledge Graph Retrieval for Behaviorally Aligned Financial Asset Recommendations](https://mlsanity.com/paper/2511.11583/) (2025)
- [SRAG: RAG with Structured Data Improves Vector Retrieval](https://mlsanity.com/paper/2603.26670/) (2026)
- [MimirRAG: A Multi-Agent RAG Framework for Financial Data Retrieval with Metadata Integration](https://mlsanity.com/paper/2605.25030/) (2026)

## Abstract (arXiv)

We propose two new methods for multi-document financial question answering. First, a method that uses semantic tagging, and then, queries the index to get the context (RAG_SEM). And second, a Knowledge Graph (KG_RAG) based method that uses semantic tagging, and, retrieves knowledge graph triples from a graph database, as context. KG_RAG uses knowledge graphs constructed using a small model that is fine-tuned using knowledge distillation using a large teacher model. The data consists of 18 10K reports of Apple, Microsoft, Alphabet, NVIDIA, Amazon and Tesla for the years 2021, 2022 and 2023. The list of questions in the data consists of 111 complex questions including many esoteric questions that are difficult to answer and the answers are not completely obvious. As evaluation metrics, we use overall scores as well as segmented scores for measurement including the faithfulness, relevance, correctness, similarity, an LLM based overall score and the rouge scores as well as a similarity of embeddings. We find that both methods outperform plain RAG significantly. KG_RAG outperforms RAG_SEM in four out of nine metrics.