mlsanityData to 7 Oct 2026

An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation

arXiv 2406.01549 ↗First posted 3 Jun 2024cs.CL cs.AIcited by 16

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-augmented generation integrates the capabilities of large language models with relevant information retrieved from an extensive corpus, yet encounters challenges when confronted with real-world noisy data. One recent solution is to train a filter module to find relevant content but only achieve suboptimal noise compression. In this paper, we propose to introduce the information bottleneck theory into retrieval-augmented generation. Our approach involves the filtration of noise by simultaneously maximizing the mutual information between compression and ground output, while minimizing the mutual information between compression and retrieved passage. In addition, we derive the formula of information bottleneck to facilitate its application in novel comprehensive evaluations, the selection of supervised fine-tuning data, and the construction of reinforcement learning rewards. Experimental results demonstrate that our approach achieves significant improvements across various question answering datasets, not only in terms of the correctness of answer generation but also in the conciseness with 2.5% compression rate.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 33 of its 47 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

33 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction24 Oct
  2. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation6 Oct
  3. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuning2 Oct
  4. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  5. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  10. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  11. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  12. 2024Large Language Models: A Surveycited by 506
15 more
  1. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  2. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  3. 2016SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documentscited by 245
  4. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  5. 2020Graph Information Bottleneckcited by 168
  6. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  7. 2020The Conditional Entropy Bottleneckcited by 75
  8. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 74
  9. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  10. 2023Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filteringcited by 62
  11. 2021Compressive Visual Representationscited by 41
  12. 2021Evidentiality-guided Generation for Knowledge-Intensive NLP Taskscited by 40
  13. 2019BottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principlecited by 39
  14. 2023PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adaptercited by 38
  15. 2023GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Rankingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 4.

16 papers.

Sort:

By area:

  1. 2025Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learningcited by 39
  2. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  3. 2025TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Modelscited by 7
  4. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  5. 2025Influence Guided Context Selection for Effective Retrieval-Augmented Generationcited by 6
  6. 2025TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottleneckscited by 4
  7. 2025Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoEcited by 2
  8. 2025Less Is More: Elevating RAG via Performance-Driven Context Compressioncited by 2
  9. 2026Experience as a Compass: Multi-agent RAG with Evolving Orchestration and Agent Promptscited by 2
  10. 2024QCG-Rerank: Chunks Graph Rerank with Query Expansion in Retrieval-Augmented LLMs for Tourism Domaincited by 1
  11. 2025Learning to Extract Rational Evidence via Reinforcement Learning for Retrieval-Augmented Generationcited by 1
  12. 2025Benchmarking Retrieval-Augmented Generation Strategies for Large Language Model-Based Travel Mode Choice Predictioncited by 1
4 more
  1. 2026Collaborative Parameter Learning: Mitigating Forgetting via Parameter-Level Gradient Analysiscited by 1
  2. 2025The Other Side of the Coin: Exploring Fairness in Retrieval-Augmented Generationcited by 0
  3. 2026GRIP: Grounded Reasoning via Information-Restricted Premisescited by 0
  4. 2026The Copy Ceiling: An Input-Exposure Control for Ontology-Grounded Generation over Curated Corporacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024QUITO-X: A New Perspective on Context Compression from the Information Bottleneck Theory
  2. 2025Noise-Robust Abstractive Compression in Retrieval-Augmented Language Models
  3. 2025ACoRN: Noise-Robust Abstractive Compression in Retrieval-Augmented Language Models
  4. 2026Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAG
  5. 2024Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation
  6. 2026AdaMem: Adaptive Memory Token Allocation for Soft Compression in Retrieval-Augmented Generation
  7. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  8. 2026Rethinking Retrieval-Augmentation as Synthesis: A Query-Aware Context Merging Approach