# HAM-RAG: Hierarchy-Aware Multimodal RAG for Structure-Faithful Interleaved Generation

arXiv: https://arxiv.org/abs/2608.14032 · HTML: https://mlsanity.com/paper/2608.14032/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 12 of its 14 references that are in the map.
- First posted on arXiv 14 Aug 2026.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 12 of 12)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)
- [VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents](https://mlsanity.com/paper/2410.10594/) (2024)
- [MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text](https://mlsanity.com/paper/2210.02928/) (2022)
- [Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs](https://mlsanity.com/paper/2404.15406/) (2024)
- [Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering](https://mlsanity.com/paper/2505.16470/) (2025)
- [MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generation](https://mlsanity.com/paper/2502.04176/) (2025)
- [Multi-modal Retrieval Augmented Multi-modal Generation: Datasets, Evaluation Metrics and Strong Baselines](https://mlsanity.com/paper/2411.16365/) (2024)
- [WikiWeb2M: A Page-Level Multimodal Wikipedia Dataset](https://mlsanity.com/paper/2305.05432/) (2023)
- [MuRAR: A Simple and Effective Multimodal Retrieval and Answer Refinement Framework for Multimodal Question Answering](https://mlsanity.com/paper/2408.08521/) (2024)
- [MLDocRAG: Multimodal Long-Context Document Retrieval Augmented Generation](https://mlsanity.com/paper/2602.10271/) (2026)

## Abstract (arXiv)

Existing multimodal RAG methods often flatten structured documents into isolated text and image units, weakening the source organization and local text-image logic needed for faithful evidence selection and placement. We propose HAM-RAG, a Hierarchy-Aware Multimodal RAG framework for structure-faithful interleaved generation. HAM-RAG uses document hierarchy as a grounding signal across retrieval and generation, contextualizing textual and visual evidence and preserving source position and local text-image relations in the prompt. We further introduce HAM-Bench, covering Wukong, Wiki, arXiv, and Recipe across game walkthroughs, web pages, scientific papers, and step-wise recipe documents. Across multiple backbones, HAM-RAG improves the main multimodal average by 17.3% over the strongest non-hierarchical baseline. On Wukong, HAM-RAG improves Img-CBS by 24.2% over the strongest non-hierarchical baseline, demonstrating substantially better local text-image alignment. The main experiments and ablation study together demonstrate that document hierarchy is a key grounding signal for faithful image selection, placement, and local text-image alignment. These findings highlight the value of hierarchy-aware grounding for reliable multimodal assistants that generate answers faithful to the source organization, procedural structure, and local text-image evidence of structured documents, such as technical manuals, maintenance guides, and industrial SOPs. The code is available at https://github.com/MCCodeAI/HAM-RAG.git.