# To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts

arXiv: https://arxiv.org/abs/2410.14675 · HTML: https://mlsanity.com/paper/2410.14675/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 33 of its 34 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 30 of 33)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [Iterative Reasoning Preference Optimization](https://mlsanity.com/paper/2404.19733/) (2024)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020)
- [Large Language Models with Controllable Working Memory](https://mlsanity.com/paper/2211.05110/) (2022)
- [Certifiably Robust RAG against Retrieval Corruption](https://mlsanity.com/paper/2405.15556/) (2024)
- [Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step](https://mlsanity.com/paper/2306.14050/) (2023)
- [On the Risk of Misinformation Pollution with Large Language Models](https://mlsanity.com/paper/2305.13661/) (2023)
- [ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence](https://mlsanity.com/paper/2404.10198/) (2024)
- [System 2 Attention (is something you might need too)](https://mlsanity.com/paper/2311.11829/) (2023)
- [FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"](https://mlsanity.com/paper/2410.03727/) (2024)
- [Reliable, Adaptable, and Attributable Language Models with Retrieval](https://mlsanity.com/paper/2403.03187/) (2024)
- [Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models](https://mlsanity.com/paper/2410.07176/) (2024)
- [DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2407.17023/) (2024)
- [Not All Contexts Are Equal: Teaching LLMs Credibility-aware Generation](https://mlsanity.com/paper/2404.06809/) (2024)
- [KnowPO: Knowledge-aware Preference Optimization for Controllable Knowledge Selection in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2408.03297/) (2024)

## Built on by (most cited first, 25 of 25)

- [Retrieval-Augmented Generation with Conflicting Evidence](https://mlsanity.com/paper/2504.13079/) (2025)
- [Leaky Thoughts: Large Reasoning Models Are Not Private Thinkers](https://mlsanity.com/paper/2506.15674/) (2025)
- [Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systems](https://mlsanity.com/paper/2601.07072/) (2026)
- [Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks](https://mlsanity.com/paper/2412.16708/) (2024)
- [The RAG Paradox: A Black-Box Attack Exploiting Unintentional Vulnerabilities in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2502.20995/) (2025)
- [A Unified Definition of Hallucination: It's The World Model, Stupid!](https://mlsanity.com/paper/2512.21577/) (2025)
- [SafeSearch: Automated Red-Teaming of LLM-Based Search Agents](https://mlsanity.com/paper/2509.23694/) (2025)
- [From Faithfulness to Correctness: Generative Reward Models that Think Critically](https://mlsanity.com/paper/2509.25409/) (2025)
- [Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Models](https://mlsanity.com/paper/2503.16581/) (2025)
- [Conflicts in Texts: Data, Implications and Challenges](https://mlsanity.com/paper/2504.19472/) (2025)
- [Know Or Not: a library for evaluating out-of-knowledge base robustness](https://mlsanity.com/paper/2505.13545/) (2025)
- [Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models](https://mlsanity.com/paper/2506.17585/) (2025)
- [Fast, Slow, and Tool-augmented Thinking for LLMs: A Review](https://mlsanity.com/paper/2508.12265/) (2025)
- [Scaling Participation in Modular AI Systems](https://mlsanity.com/paper/2606.07812/) (2026)
- [When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs](https://mlsanity.com/paper/2510.09106/) (2025)
- [Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning](https://mlsanity.com/paper/2602.14518/) (2026)
- [Trust or Abstain? A Self-Aware RAG Approach](https://mlsanity.com/paper/2605.18792/) (2026)
- [Does a Tool Result Carry More Authority Than Plain Text? Three Prospective Studies of False-Claim Adoption in a Synthetic Assignment Task with Claude Opus 5](https://mlsanity.com/paper/2608.14992/) (2026)
- [Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generation](https://mlsanity.com/paper/2608.24977/) (2026)
- [Return or Revise? Learning When Revision Helps Retrieval-Augmented QA](https://mlsanity.com/paper/2609.30087/) (2026)
- [Can LLMs Take Retrieved Information with a Grain of Salt?](https://mlsanity.com/paper/2605.06919/) (2026)
- [Faithful by Definition: Emotion Analysis via Natural Semantic Metalanguage Explications](https://mlsanity.com/paper/2607.00661/) (2026)
- [When Context Misleads: Intent-Guided Decoding for Robust Retrieval-Augmented Generation](https://mlsanity.com/paper/2608.16515/) (2026)
- [Lazy Grounding: Attacking Search Agents with Factual Evidence](https://mlsanity.com/paper/2608.30303/) (2026)
- [The Answer Path and the Grounding Instruction in LLM Question Answering over Knowledge Graphs](https://mlsanity.com/paper/2609.10237/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are often augmented with external contexts, such as those used in retrieval-augmented generation (RAG). However, these contexts can be inaccurate or intentionally misleading, leading to conflicts with the model's internal knowledge. We argue that robust LLMs should demonstrate situated faithfulness, dynamically calibrating their trust in external information based on their confidence in the internal knowledge and the external context to resolve knowledge conflicts. To benchmark this capability, we evaluate LLMs across several QA datasets, including a newly created dataset featuring in-the-wild incorrect contexts sourced from Reddit posts. We show that when provided with both correct and incorrect contexts, both open-source and proprietary models tend to overly rely on external information, regardless of its factual accuracy. To enhance situated faithfulness, we propose two approaches: Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR). SCR enables models to self-assess the confidence of external information relative to their own internal knowledge to produce the most accurate answer. RCR, in contrast, extracts explicit confidence signals from the LLM and determines the final answer using predefined rules. Our results show that for LLMs with strong reasoning capabilities, such as GPT-4o and GPT-4o mini, SCR outperforms RCR, achieving improvements of up to 24.2% over a direct input augmentation baseline. Conversely, for a smaller model like Llama-3-8B, RCR outperforms SCR. Fine-tuning SCR with our proposed Confidence Reasoning Direct Preference Optimization (CR-DPO) method improves performance on both seen and unseen datasets, yielding an average improvement of 8.9% on Llama-3-8B. In addition to quantitative results, we offer insights into the relative strengths of SCR and RCR.