mlsanityData to 13 Oct 2026

Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models

arXiv 2501.01059 ↗First posted 2 Jan 2025cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) often exhibit Context Faithfulness Hallucinations, where outputs deviate from retrieved information due to incomplete context integration. Our analysis reveals a strong correlation between token-level uncertainty and hallucinations. We hypothesize that attention mechanisms inherently encode context utilization signals, supported by probing analysis. Based on these insights, we propose Dynamic Attention-Guided Context Decoding (DAGCD), a lightweight framework that leverages attention distributions and uncertainty signals in a single-pass decoding. Experiments on open-book QA datasets demonstrate DAGCD's effectiveness, yielding significant improvements in faithfulness and robustness while preserving computational efficiency.

Key facts. Cited by 9 papers in the map (data through 13 Oct 2026). Builds on 44 of its 52 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

44 of 52 references.

References · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,480
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,716
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,925
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,650
  9. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,371
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,196
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,131
32 more
  1. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,288
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 999
  3. 2019Attention is not Explanationcited by 922
  4. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 600
  5. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 517
  6. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 352
  7. 2023Explainability for Large Language Models: A Surveycited by 333
  8. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  9. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 324
  10. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 312
  11. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 242
  12. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 224
  13. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 222
  14. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  15. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  16. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 141
  17. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  18. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 125
  19. 2024Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Modelscited by 93
  20. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 88
  21. 2019Attention Interpretability Across NLP Taskscited by 71
  22. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  23. 2024Attention Heads of Large Language Models: A Surveycited by 67
  24. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  25. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  26. 2023KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detectioncited by 45
  27. 2024IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying and Reweighting Context-Aware Neuronscited by 37
  28. 2024AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledgecited by 36
  29. 2020Understanding Neural Abstractive Summarization Models via Uncertaintycited by 34
  30. 2024Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantizationcited by 33
  31. 2024Discerning and Resolving Knowledge Conflicts through Adaptive Decoding with Contextual Information-Entropy Constraintcited by 27
  32. 2024Hallucination Detection and Hallucination Mitigation: An Investigationcited by 20

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  2. 2026Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Modelscited by 4
  3. 2026Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMscited by 3
  4. 2025Beyond Black-Box Interventions: Latent Probing for Faithful Retrieval-Augmented Generationcited by 2
  5. 2025COMPASS: Context-Modulated PID Attention Steering System for Hallucination Mitigationcited by 2
  6. 2026Detecting Contextual Hallucinations in LLMs with Frequency-Aware Attentioncited by 2
  7. 2025HAVE: Head-Adaptive Gating and ValuE Calibration for Hallucination Mitigation in Large Language Modelscited by 0
  8. 2026On Improving Faithfulness of Podcasts from Documentscited by 0
  9. 2026Decoupling Generation and Selection for Budget-Constrained Faithful Summarizationcited by 0