mlsanityData to 7 Oct 2026

Entropy-Based Decoding for Retrieval-Augmented Large Language Models

arXiv 2406.17519 ↗First posted 25 Jun 2024cs.CLcited by 19

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Augmenting Large Language Models (LLMs) with retrieved external knowledge has proven effective for improving the factual accuracy of generated responses. Despite their success, retrieval-augmented LLMs still face the distractibility issue, where the generated responses are negatively influenced by noise from both external and internal knowledge sources. In this paper, we introduce a novel, training-free decoding method guided by entropy considerations to mitigate this issue. Our approach utilizes entropy-based document-parallel ensemble decoding to prioritize low-entropy distributions from retrieved documents, thereby enhancing the extraction of relevant information of context. Additionally, it incorporates a contrastive decoding mechanism that contrasts the obtained low-entropy ensemble distribution with the high-entropy distribution derived from the model's internal knowledge across layers, which ensures a greater emphasis on reliable external information. Extensive experiments on open-domain question answering datasets demonstrate the superiority of our method.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 34 of its 41 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

34 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuning2 Oct
  2. 2023In-Context Retrieval-Augmented Language Models31 Jan
  3. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  4. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories20 Dec
  5. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  10. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  11. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
15 more
  1. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  3. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  4. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  5. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  6. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  7. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  8. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  9. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  10. 2022RealTime QA: What's the Answer Right Now?cited by 239
  11. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
  12. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  13. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 71
  14. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  15. 2024CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 10.

19 papers.

Sort:

By area:

  1. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  2. 2024Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generationcited by 63
  3. 2025Mark Your LLM: Detecting the Misuse of Open-Source Large Language Models via Watermarkingcited by 14
  4. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  5. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decodingcited by 12
  6. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4
  7. 2026Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibrationcited by 4
  8. 2025Bridging External and Parametric Knowledge: Mitigating Hallucination of LLMs with Shared-Private Semantic Synergy in Dual-Stream Knowledgecited by 3
  9. 2025Entropy-Aware Token Rejection for Improving Speculative Decodingcited by 2
  10. 2025Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decodingcited by 1
  11. 2026Parallel Context-of-Experts Decoding for Retrieval Augmented Generationcited by 1
  12. 2026When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMscited by 1
7 more
  1. 2026CompactRAG: Reducing LLM Calls and Token Overhead in Multi-Hop Question Answeringcited by 1
  2. 2026Entropy-Guided k-Guard Sampling for Long-Horizon Autoregressive Video Generationcited by 0
  3. 2026BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answeringcited by 0
  4. 2026Uncertainty Quantification for Multimodal Retrieval Augmented Generationcited by 0
  5. 2026TrustMargin: Training-Free Arbitration between Parametric Memory and Retrieved Evidence in Large Language Modelscited by 0
  6. 2026LODESTAR: Robust Entropy-Based Answer Selection in Retrieval-Augmented Generation for Question Answering -- Directing Frozen-LLM Entropy with a Reinforcement-Learned Prompt Polarizer under Misleading Passagescited by 0
  7. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts
  2. 2026Think Twice Before You Write -- an Entropy-based Decoding Strategy to Enhance LLM Reasoning
  3. 2025BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generation
  4. 2026Beyond the Context Window: An Adaptive Entropy-Based Routing Framework for Hybrid Retrieval and Long-Context Language Models
  5. 2025LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generation
  6. 2024Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models
  7. 2026Entropy-Aligned Decoding of LMs for Better Writing and Reasoning
  8. 2024Unraveling and Mitigating Retriever Inconsistencies in Retrieval-Augmented Large Language Models