# Multimodal RAG for Unstructured Data:Leveraging Modality-Aware Knowledge Graphs with Hybrid Retrieval

arXiv: https://arxiv.org/abs/2510.14592 · HTML: https://mlsanity.com/paper/2510.14592/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 7 of its 17 references that are in the map.
- First posted on arXiv 16 Oct 2025.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 7 of 7)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Knowledge Graph-Guided Retrieval Augmented Generation](https://mlsanity.com/paper/2502.06864/) (2025)
- [UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis](https://mlsanity.com/paper/2406.15187/) (2024)
- [REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark](https://mlsanity.com/paper/2502.12342/) (2025)
- [MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation](https://mlsanity.com/paper/2502.04176/) (2025)
- [DO-RAG: A Domain-Specific QA Framework Using Knowledge Graph-Enhanced Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.17058/) (2025)

## Built on by (most cited first, 2 of 2)

- [Task-Adaptive Retrieval over Agentic Multi-Modal Web Histories via Learned Graph Memory](https://mlsanity.com/paper/2604.07863/) (2026)
- [Geometry-Aware Localized Watermarking for Copyright Protection in Embedding-as-a-Service](https://mlsanity.com/paper/2604.11344/) (2026)

## Abstract (arXiv)

Current Retrieval-Augmented Generation (RAG) systems primarily operate on unimodal textual data, limiting their effectiveness on unstructured multimodal documents. Such documents often combine text, images, tables, equations, and graphs, each contributing unique information. In this work, we present a Modality-Aware Hybrid retrieval Architecture (MAHA), designed specifically for multimodal question answering with reasoning through a modality-aware knowledge graph. MAHA integrates dense vector retrieval with structured graph traversal, where the knowledge graph encodes cross-modal semantics and relationships. This design enables both semantically rich and context-aware retrieval across diverse modalities. Evaluations on multiple benchmark datasets demonstrate that MAHA substantially outperforms baseline methods, achieving a ROUGE-L score of 0.486, providing complete modality coverage. These results highlight MAHA's ability to combine embeddings with explicit document structure, enabling effective multimodal retrieval. Our work establishes a scalable and interpretable retrieval framework that advances RAG systems by enabling modality-aware reasoning over unstructured multimodal data.