mlsanityData to 7 Oct 2026

Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding

arXiv 2405.02750 ↗First posted 4 May 2024cs.CL cs.AIcited by 49

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) tend to inadequately integrate input context during text generation, relying excessively on encoded prior knowledge in model parameters, potentially resulting in generated text with factual inconsistencies or contextually unfaithful content. LLMs utilize two primary knowledge sources: 1) prior (parametric) knowledge from pretraining, and 2) contextual (non-parametric) knowledge from input prompts. The study addresses the open question of how LLMs effectively balance these knowledge sources during the generation process, specifically in the context of open-domain question answering. To address this issue, we introduce a novel approach integrating contrastive decoding with adversarial irrelevant passages as negative samples to enhance robust context grounding during generation. Notably, our method operates at inference time without requiring further training. We conduct comprehensive experiments to demonstrate its applicability and effectiveness, providing empirical evidence showcasing its superiority over existing methodologies. Our code is publicly available at: https://github.com/amazon-science/ContextualUnderstanding-ContrastiveDecoding.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 4 May 2024.

Builds on

29 of 40 references.

References · 29

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  10. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  11. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  12. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
17 more
  1. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  2. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  3. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  4. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  5. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  6. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  7. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  8. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  9. 2022RealTime QA: What's the Answer Right Now?cited by 239
  10. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
  11. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  12. 2022Training Language Models with Memory Augmentationcited by 112
  13. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  14. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  15. 2021Coherence boosting: When your pretrained language model is not paying enough attentioncited by 38
  16. 2023Merging Generated and Retrieved Knowledge for Open-Domain QAcited by 36
  17. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 25; 2026: 13.

49 papers.

Sort:

By area:

  1. 2024Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Modelscited by 57
  2. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  3. 2024AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledgecited by 35
  4. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  5. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  6. 2024DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinationscited by 23
  7. 2024Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decodingcited by 20
  8. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  9. 2024Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contextscited by 19
  10. 2025SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidencecited by 18
  11. 2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13
  12. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decodingcited by 12
37 more
  1. 2025Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributionscited by 11
  2. 2025Evaluating the Sensitivity of LLMs to Prior Contextcited by 9
  3. 2025CoCoA: Confidence and Context-Aware Adaptive Decoding for Resolving Knowledge Conflicts in Large Language Modelscited by 9
  4. 2024When to Speak, When to Abstain: Contrastive Decoding with Abstentioncited by 7
  5. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  6. 2025CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generationcited by 6
  7. 2026Gated Differentiable Working Memory for Long-Context Language Modelingcited by 6
  8. 2025Conflict-Aware Soft Prompting for Retrieval-Augmented Generationcited by 5
  9. 2026Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoningcited by 5
  10. 2024DEXTER: A Benchmark for open-domain Complex Question Answering using LLMscited by 4
  11. 2024Multilingual Contrastive Decoding via Language-Agnostic Layers Skippingcited by 4
  12. 2025Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decodingcited by 4
  13. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4
  14. 2026Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibrationcited by 4
  15. 2025Reimagining Urban Science: Scaling Causal Inference with Large Language Modelscited by 3
  16. 2025CUB: Benchmarking Context Utilisation Techniques for Language Modelscited by 3
  17. 2026ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Modelscited by 3
  18. 2025Model Editing with Graph-Based External Memorycited by 2
  19. 2025Dual Debiasing for Noisy In-Context Learning for Text Generationcited by 2
  20. 2025COMPASS: Context-Modulated PID Attention Steering System for Hallucination Mitigationcited by 2
  21. 2026CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answeringcited by 2
  22. 2025Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Modelscited by 1
  23. 2025SynDec: A Synthesize-then-Decode Approach for Arbitrary Textual Style Transfer via Large Language Modelscited by 1
  24. 2025Toward Inherently Robust VLMs Against Visual Perception Attackscited by 1
  25. 2025Multi-Step Knowledge Interaction Analysis via Rank-2 Subspace Disentanglementcited by 1
  26. 2025Short-Context Dominance: How Much Local Context Natural Language Actually Needs?cited by 1
  27. 2026Decoding in Geometry: Alleviating Embedding-Space Crowding for Complex Reasoningcited by 1
  28. 2025UniKnow: A Unified Framework for Reliable Language Model Behavior across Parametric and External Knowledgecited by 0
  29. 2025Performance Gains of LLMs With Humans in a World of LLMs Versus Humanscited by 0
  30. 2025MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generationcited by 0
  31. 2026CE-GOCD: Central Entity-Guided Graph Optimization for Community Detection to Augment LLM Scientific Question Answeringcited by 0
  32. 2026Dual-Confidence Contrastive Decoding for Retrieval-Augmented Generationcited by 0
  33. 2026Development of Vision-Language Model-based GNSS Spoofing Detection for Autonomous Vehicle Navigationcited by 0
  34. 2026Decoupled Contrastive Decoding via Expert-Aligned Draftingcited by 0
  35. 2026Can Language Models Understand mmWave Data? Benchmarking Large Language Models for mmWave Radar-Based Human Understandingcited by 0
  36. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0
  37. 2026RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptationcited by 0