# Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models

arXiv: https://arxiv.org/abs/2501.01059 · HTML: https://mlsanity.com/paper/2501.01059/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 52 references that are in the map.
- First posted on arXiv 2 Jan 2025.

## Builds on (most cited first, 30 of 44)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [Analyzing the Structure of Attention in a Transformer Language Model](https://mlsanity.com/paper/1906.04284/) (2019)
- [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples](https://mlsanity.com/paper/2310.01469/) (2023)
- [Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps](https://mlsanity.com/paper/2407.07071/) (2024)
- [Understanding and Overcoming the Challenges of Efficient Transformer Quantization](https://mlsanity.com/paper/2109.12948/) (2021)
- [Self-Attention Attribution: Interpreting Information Interactions Inside Transformer](https://mlsanity.com/paper/2004.11207/) (2020)

## Built on by (most cited first, 9 of 9)

- [Large Language Models Hallucination: A Comprehensive Survey](https://mlsanity.com/paper/2510.06265/) (2025)
- [Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models](https://mlsanity.com/paper/2602.05897/) (2026)
- [Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs](https://mlsanity.com/paper/2602.15556/) (2026)
- [Beyond Black-Box Interventions: Latent Probing for Faithful Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.12460/) (2025)
- [COMPASS: Context-Modulated PID Attention Steering System for Hallucination Mitigation](https://mlsanity.com/paper/2511.14776/) (2025)
- [Detecting Contextual Hallucinations in LLMs with Frequency-Aware Attention](https://mlsanity.com/paper/2602.18145/) (2026)
- [HAVE: Head-Adaptive Gating and ValuE Calibration for Hallucination Mitigation in Large Language Models](https://mlsanity.com/paper/2509.06596/) (2025)
- [On Improving Faithfulness of Podcasts from Documents](https://mlsanity.com/paper/2607.21961/) (2026)
- [Decoupling Generation and Selection for Budget-Constrained Faithful Summarization](https://mlsanity.com/paper/2608.03655/) (2026)

## Abstract (arXiv)

Large language models (LLMs) often exhibit Context Faithfulness Hallucinations, where outputs deviate from retrieved information due to incomplete context integration. Our analysis reveals a strong correlation between token-level uncertainty and hallucinations. We hypothesize that attention mechanisms inherently encode context utilization signals, supported by probing analysis. Based on these insights, we propose Dynamic Attention-Guided Context Decoding (DAGCD), a lightweight framework that leverages attention distributions and uncertainty signals in a single-pass decoding. Experiments on open-book QA datasets demonstrate DAGCD's effectiveness, yielding significant improvements in faithfulness and robustness while preserving computational efficiency.