mlsanityData to 7 Oct 2026

Parallel Key-Value Cache Fusion for Position Invariant RAG

arXiv 2501.07523 ↗First posted 13 Jan 2025cs.AI cs.CLcited by 0

On the map: Retrieval-augmented generation → Generation and context integration

also in 1 more
Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) underscore the necessity of Retrieval Augmented Generation (RAG) to leverage external information. However, LLMs are sensitive to the position of relevant information within contexts and tend to generate incorrect responses when such information is placed in the middle, known as `Lost in the Middle' phenomenon. In this paper, we introduce a framework that generates consistent outputs for decoder-only models, irrespective of the input context order. Experimental results for three open domain question answering tasks demonstrate position invariance, where the model is not sensitive to input context order, and superior robustness to irrelevent passages compared to prevailing approaches for RAG pipelines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 34 of its 44 references that are in the map. First posted on arXiv 13 Jan 2025.

Builds on

34 of 44 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks20 Dec
  2. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs2 Jul
  3. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation6 Oct
  4. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context2 Oct
  5. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  6. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories20 Dec
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  8. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
13 more
  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  3. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  4. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
  5. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  6. 2022Teaching language models to support answers with verified quotescited by 290
  7. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  8. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  9. 2024Make Your LLM Fully Utilize the Contextcited by 118
  10. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 97
  11. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Trainingcited by 88
  12. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  13. 2023Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Trainingcited by 31

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025REFRAG: Rethinking RAG based Decoding
  2. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  3. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
  4. 2024Toward Optimal Search and Retrieval for RAG
  5. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation
  6. 2026Knowing Before Answering: Decoding Language Models for Reliable RAG
  7. 2024Lighter And Better: Towards Flexible Context Adaptation For Retrieval Augmented Generation
  8. 2025Optimizing Retrieval-Augmented Generation: Analysis of Hyperparameter Impact on Performance and Efficiency