mlsanityData to 14 Oct 2026

MHA-RAG: Improving Efficiency, Accuracy, and Consistency by Encoding Exemplars as Soft Prompts

arXiv 2510.05363 ↗First posted 6 Oct 2025cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Adapting Foundation Models to new domains with limited training data is challenging and computationally expensive. While prior work has demonstrated the effectiveness of using domain-specific exemplars as in-context demonstrations, we investigate whether representing exemplars purely as text is the most efficient, effective, and stable approach. We explore an alternative: representing exemplars as soft prompts with an exemplar order invariant model architecture. To this end, we introduce Multi-Head Attention Retrieval-Augmented Generation (MHA-RAG), a framework with the number of attention heads serving as a simple hyperparameter to control soft prompt-generation across different tasks. Across multiple question-answering benchmarks and model scales, MHA-RAG achieves a 20-point performance gain over standard RAG, while cutting inference costs by a factor of 10X GFLOPs-delivering both higher accuracy and greater efficiency, invariant to exemplar order.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 47 of its 56 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

47 of 56 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
35 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  6. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  8. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  11. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  12. 2021GPT Understands, Toocited by 716
  13. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  14. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  15. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  16. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  17. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  18. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  19. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 185
  20. 2022Translation between Molecules and Natural Languagecited by 177
  21. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  22. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  23. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  24. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 127
  25. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 125
  26. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
  27. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 72
  28. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directionscited by 70
  29. 2022BBTv2: Towards a Gradient-Free Future with Large Language Modelscited by 53
  30. 2022IDPG: An Instance-Dependent Prompt Generation Methodcited by 43
  31. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32
  32. 2025Solving Inequality Proofs with Large Language Modelscited by 21
  33. 2023In-Context Exemplars as Clues to Retrieving from Large Associative Memorycited by 13
  34. 2024Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptationcited by 8
  35. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systemscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating Prompt Engineering Techniques for RAG in Small Language Models: A Multi-Hop QA Approach
  2. 2024GEM-RAG: Graphical Eigen Memories For Retrieval Augmented Generation
  3. 2026When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs
  4. 2024Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
  5. 2025Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMs
  6. 2024From RAGs to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queries
  7. 2024Toward Optimal Search and Retrieval for RAG
  8. 2024W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering