# Improving Contextual Faithfulness of Large Language Models via Retrieval Heads-Induced Optimization

arXiv: https://arxiv.org/abs/2501.13573 · HTML: https://mlsanity.com/paper/2501.13573/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- First posted on arXiv 23 Jan 2025.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Built on by (most cited first, 17 of 17)

- [Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking](https://mlsanity.com/paper/2506.09944/) (2025)
- [ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.15543/) (2025)
- [Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning](https://mlsanity.com/paper/2505.16483/) (2025)
- [From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models](https://mlsanity.com/paper/2601.11020/) (2026)
- [FaithRL: Learning to Reason Faithfully through Step-Level Faithfulness Maximization](https://mlsanity.com/paper/2602.03507/) (2026)
- [Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models](https://mlsanity.com/paper/2508.18381/) (2025)
- [FaithLens: Detecting and Explaining Faithfulness Hallucination](https://mlsanity.com/paper/2512.20182/) (2025)
- [DaPT: A Dual-Path Framework for Multilingual Multi-hop Question Answering](https://mlsanity.com/paper/2603.19097/) (2026)
- [SHIFT: Gate-Modulated Activation Steering for Knowledge Conflict Mitigation in Retrieval-Augmented Generation](https://mlsanity.com/paper/2606.27786/) (2026)
- [Copy-Paste to Mitigate Large Language Model Hallucinations](https://mlsanity.com/paper/2510.00508/) (2025)
- [Beyond Black-Box Interventions: Latent Probing for Faithful Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.12460/) (2025)
- [DIAL-KG: Schema-Free Incremental Knowledge Graph Construction via Dynamic Schema Induction and Evolution-Intent Assessment](https://mlsanity.com/paper/2603.20059/) (2026)
- [When LLMs Read Tables Carelessly: Measuring and Reducing Data Referencing Errors](https://mlsanity.com/paper/2606.32029/) (2026)
- [One for All: Update Parameterized Knowledge Across Multiple Models](https://mlsanity.com/paper/2506.00817/) (2025)
- [Enhancing Trustworthy Clinical Diagnosis Decision-Making in Large Language Models via Etiology-Aware Attention Supervision](https://mlsanity.com/paper/2508.00285/) (2025)
- [Identifying Crucial Attention Heads for Multilingual Language Models: Retrieval and Retrieval-Transition Heads](https://mlsanity.com/paper/2602.22453/) (2026)
- [Better Decomposition, Free Aggregation: A Synthesizer-Folding Framework for Multilingual Multi-Hop Question Answering](https://mlsanity.com/paper/2608.13160/) (2026)

## Abstract (arXiv)

Ensuring contextual faithfulness in retrieval-augmented large language models (LLMs) is crucial for building trustworthy information-seeking systems, particularly in long-form question-answering (LFQA) scenarios. In this work, we identify a salient correlation between LFQA faithfulness and retrieval heads, a set of attention heads responsible for retrieving contextual information. Leveraging this insight, we propose RHIO, a framework designed to teach LLMs to explicitly discriminate between faithful and unfaithful generations. RHIO first augments unfaithful samples that simulate realistic model-intrinsic errors by selectively masking retrieval heads. Then, these samples are incorporated into joint training, enabling the model to distinguish unfaithful outputs from faithful ones conditioned on control tokens. Furthermore, these control tokens are leveraged to self-induce contrastive outputs, amplifying their difference through contrastive decoding. Additionally, to facilitate the evaluation of contextual faithfulness, we also introduce GroundBench, a comprehensive benchmark compiled from five existing LFQA datasets. Extensive experimental results on GroundBench demonstrate that RHIO significantly improves faithfulness, even outperforming GPT-4o.