mlsanityData to 7 Oct 2026

Multi-Head RAG: Solving Multi-Aspect Problems with LLMs

arXiv 2406.05085 ↗First posted 7 Jun 2024cs.CL cs.AI cs.IRcited by 18

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) improves Large Language Models (LLMs) by retrieving supporting documents into the prompt, but existing methods do not explicitly target queries that require fetching multiple documents with substantially different content. Such multi-aspect queries are challenging because relevant documents can be far apart in embedding space, making joint retrieval difficult. We introduce Multi-Head RAG (MRAG), which addresses this gap with a simple yet powerful idea: using Transformer multi-head attention activations rather than the standard decoder-layer embedding, as retrieval keys. It leverages the observation that different heads capture different semantic aspects. This yields multi-aspect embeddings for both documents and queries, improving retrieval accuracy on complex queries. We show MRAG's design advantages over 18 RAG baselines, up to 20% higher retrieval success ratios for real-world use cases, and improved downstream LLM generation. MRAG integrates seamlessly with existing RAG frameworks and benchmarks.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 32 of its 41 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

32 of 41 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Evaluation of Retrieval-Augmented Generation: A Survey13 May
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Survey29 Feb
  3. 2024RAG-Fusion: a New Take on Retrieval-Augmented Generation31 Jan
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  5. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation26 Sep
  6. 2022A Survey on Retrieval-Augmented Text Generation2 Feb

Other references · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  5. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  6. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  7. 2019Passage Re-ranking with BERTcited by 629
  8. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  9. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  10. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  11. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  12. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 180
14 more
  1. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  2. 2024Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answeringcited by 95
  3. 2024On Protecting the Data Privacy of Large Language Models (LLMs): A Surveycited by 91
  4. 2019CEDR: Contextualized Embeddings for Document Rankingcited by 77
  5. 2021Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipelinecited by 65
  6. 2020PARADE: Passage Representation Aggregation for Document Rerankingcited by 44
  7. 2024History, Development, and Principles of Large Language Models-An Introductory Surveycited by 30
  8. 2023HyKGE: A Hypothesis Knowledge Graph Enhanced Framework for Accurate and Reliable Medical LLMs Responsescited by 25
  9. 2022In Defense of Cross-Encoders for Zero-Shot Retrievalcited by 21
  10. 2024Graph-Based Retriever Captures the Long Tail of Biomedical Knowledgecited by 20
  11. 2024Federated Recommendation via Hybrid Retrieval Augmented Generationcited by 16
  12. 2024Cross-Data Knowledge Graph Construction for LLM-enabled Educational Question-Answering System: A Case Study at HCMUTcited by 14
  13. 2023Retrieval Augmented Generation and Representative Vector Summarization for large unstructured textual data in Medical Educationcited by 10
  14. 2021Updating Embeddings for Dynamic Knowledge Graphscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 1.

18 papers.

Sort:

By area:

  1. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  2. 2024Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworkscited by 41
  3. 2025Reasoning Language Models: A Blueprintcited by 41
  4. 2025DualRAG: A Dual-Process Approach to Integrate Reasoning and Retrieval for Multi-Hop Question Answeringcited by 20
  5. 2025MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation Systemcited by 19
  6. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 18
  7. 2025FinSage: A Multi-aspect RAG System for Financial Filings Question Answeringcited by 18
  8. 2024Meta-Chunking: Learning Text Segmentation and Semantic Completion via Logical Perceptioncited by 17
  9. 2025SuperRAG: Beyond RAG with Layout-Aware Graph Modelingcited by 9
  10. 2025Hierarchical Lexical Graph for Enhanced Multi-Hop Retrievalcited by 7
  11. 2025Affordable AI Assistants with Knowledge Graph of Thoughtscited by 6
  12. 2025A Survey on Proactive Defense Strategies Against Misinformation in Large Language Modelscited by 5
6 more
  1. 2025Securing RAG: A Risk Assessment and Mitigation Frameworkcited by 3
  2. 2025When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using GradNormIRcited by 3
  3. 2025SymbioticRAG: Enhancing Document Intelligence Through Human-LLM Symbiotic Collaborationcited by 1
  4. 2025Psychologically Enhanced AI Agentscited by 1
  5. 2025Think Parallax: Solving Multi-Hop Problems via Multi-View Knowledge-Graph-Based Retrieval-Augmented Generationcited by 1
  6. 2026Performance Foundations of Parallel & Distributed Reasoning Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Retrieval-augmented generation in multilingual settings
  2. 2025Re-ranking the Context for Multimodal Retrieval Augmented Generation
  3. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models
  4. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  5. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
  6. 2025Transforming Questions and Documents for Semantically Aligned Retrieval-Augmented Generation
  7. 2025Less LLM, More Documents: Searching for Improved RAG
  8. 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practices