# CrAM: Credibility-Aware Attention Modification in LLMs for Combating Misinformation in RAG

arXiv: https://arxiv.org/abs/2406.11497 · HTML: https://mlsanity.com/paper/2406.11497/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 30 of its 50 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Certifiably Robust RAG against Retrieval Corruption](https://mlsanity.com/paper/2405.15556/) (2024)
- [Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2402.14409/) (2024)
- [Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise](https://mlsanity.com/paper/2305.01579/) (2023)
- [Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4](https://mlsanity.com/paper/2305.14928/) (2023)
- [Attacking Open-domain Question Answering by Injecting Misinformation](https://mlsanity.com/paper/2110.07803/) (2021)
- [AMMeBa: A Large-Scale Survey and Dataset of Media-Based Misinformation In-The-Wild](https://mlsanity.com/paper/2405.11697/) (2024)
- [Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generation](https://mlsanity.com/paper/2404.06809/) (2024)
- [Interpretable Multimodal Misinformation Detection with Logic Reasoning](https://mlsanity.com/paper/2305.05964/) (2023)
- [Do Sentence Interactions Matter? Leveraging Sentence Level Representations for Fake News Classification](https://mlsanity.com/paper/1910.12203/) (2019)
- [Defending Against Disinformation Attacks in Open-Domain Question Answering](https://mlsanity.com/paper/2212.10002/) (2022)
- [The Future of Misinformation Detection: New Perspectives and Trends](https://mlsanity.com/paper/1909.03654/) (2019)

## Built on by (most cited first, 22 of 22)

- [Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2502.06872/) (2025)
- [Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs](https://mlsanity.com/paper/2408.12060/) (2024)
- [Retrieval-Augmented Generation with Estimation of Source Reliability](https://mlsanity.com/paper/2410.22954/) (2024)
- [FunnelRAG: A Coarse-to-Fine Progressive Retrieval Paradigm for RAG](https://mlsanity.com/paper/2410.10293/) (2024)
- [ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Search](https://mlsanity.com/paper/2509.23519/) (2025)
- [Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment](https://mlsanity.com/paper/2502.14354/) (2025)
- [ArgRAG: Explainable Retrieval Augmented Generation using Quantitative Bipolar Argumentation](https://mlsanity.com/paper/2508.20131/) (2025)
- [OpenDecoder: Open Large Language Model Decoding to Incorporate Document Quality in RAG](https://mlsanity.com/paper/2601.09028/) (2026)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Source Coverage and Citation Bias in LLM-based vs. Traditional Search Engines](https://mlsanity.com/paper/2512.09483/) (2025)
- [Memory Helps, but Confabulation Misleads: Understanding Streaming Events in Videos with MLLMs](https://mlsanity.com/paper/2502.15457/) (2025)
- [GETReason: Enhancing Image Context Extraction through Hierarchical Multi-Agent Reasoning](https://mlsanity.com/paper/2505.21863/) (2025)
- [Tackling the Inherent Difficulty of Noise Filtering in RAG](https://mlsanity.com/paper/2601.01896/) (2026)
- [RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systems](https://mlsanity.com/paper/2601.12991/) (2026)
- [Misinformation Propagation in Benign Multi-Agent Systems](https://mlsanity.com/paper/2606.16710/) (2026)
- [AuthorityBench: Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.25092/) (2026)
- [CrEst: Credibility Estimation for Contexts in LLMs via Weak Supervision](https://mlsanity.com/paper/2506.14912/) (2025)
- [RAGSieve: Self-Referenced Local Contrast for Knowledge-Poison Detection in Retrieval-Augmented Generation](https://mlsanity.com/paper/2608.13010/) (2026)
- [RAGSentinel: Certifiable Geometric Consensus for Robust Retrieval-Augmented Generation](https://mlsanity.com/paper/2608.23965/) (2026)
- [CiteShade: Citation Laundering in Multi-Source Retrieval-Augmented Generation and Its Counterfactual Defense](https://mlsanity.com/paper/2609.15660/) (2026)
- [BaRe-Mem: Bayesian Reliability Memory for Robust and Adaptive Agent Consultation](https://mlsanity.com/paper/2609.35551/) (2026)
- [Multi-Channel Mitigation of Source-Trust Shortcuts in Fact-Checking RL Agents](https://mlsanity.com/paper/2609.36611/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) can alleviate hallucinations of Large Language Models (LLMs) by referencing external documents. However, the misinformation in external documents may mislead LLMs' generation. To address this issue, we explore the task of "credibility-aware RAG", in which LLMs automatically adjust the influence of retrieved documents based on their credibility scores to counteract misinformation. To this end, we introduce a plug-and-play method named Credibility-aware Attention Modification (CrAM). CrAM identifies influential attention heads in LLMs and adjusts their attention weights based on the credibility of the documents, thereby reducing the impact of low-credibility documents. Experiments on Natual Questions and TriviaQA using Llama2-13B, Llama3-8B, and Qwen1.5-7B show that CrAM improves the RAG performance of LLMs against misinformation pollution by over 20%, even surpassing supervised fine-tuning methods.