# Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence

arXiv: https://arxiv.org/abs/2601.11886 · HTML: https://mlsanity.com/paper/2601.11886/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 17 Jan 2026.

## Builds on (most cited first, 23 of 23)

- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence](https://mlsanity.com/paper/2210.13701/) (2022)
- [On the Risk of Misinformation Pollution with Large Language Models](https://mlsanity.com/paper/2305.13661/) (2023)
- [The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation](https://mlsanity.com/paper/2312.09085/) (2023)
- [DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering](https://mlsanity.com/paper/2211.05655/) (2022)
- [ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM](https://mlsanity.com/paper/2408.12076/) (2024)
- [Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes](https://mlsanity.com/paper/2309.00237/) (2023)
- [CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs](https://mlsanity.com/paper/2505.11413/) (2025)
- [Who's Who: Large Language Models Meet Knowledge Conflicts in Practice](https://mlsanity.com/paper/2410.15737/) (2024)
- [Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews](https://mlsanity.com/paper/2305.11828/) (2023)
- [Task Matters: Knowledge Requirements Shape LLM Responses to Context-Memory Conflict](https://mlsanity.com/paper/2506.06485/) (2025)
- [WinoDict: Probing language models for in-context word acquisition](https://mlsanity.com/paper/2209.12153/) (2022)
- [RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media](https://mlsanity.com/paper/2210.06331/) (2022)
- [Do Multi-Document Summarization Models Synthesize?](https://mlsanity.com/paper/2301.13844/) (2023)
- [Can Large Language Models Match the Conclusions of Systematic Reviews?](https://mlsanity.com/paper/2505.22787/) (2025)
- [How does Misinformation Affect Large Language Model Behaviors and Preferences?](https://mlsanity.com/paper/2505.21608/) (2025)
- [Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicine](https://mlsanity.com/paper/2506.20876/) (2025)

## Built on by (most cited first, 3 of 3)

- [What's in a Name? Morphological Shortcuts by LLMs in Pharmacology](https://mlsanity.com/paper/2606.05616/) (2026)
- [CDEG: Learning Decision-Critical Evidence for Long-Horizon Diagnostic Agents](https://mlsanity.com/paper/2608.22899/) (2026)
- [Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparities](https://mlsanity.com/paper/2610.00606/) (2026)

## Abstract (arXiv)

In high-stakes domains like medicine, it may be generally desirable for models to faithfully adhere to the context provided. But what happens if the context does not align with model priors or safety protocols? In this paper, we investigate how LLMs behave and reason when presented with counterfactual (or even adversarial) medical evidence. We first construct MedCounterFact, a counterfactual medical QA dataset that requires the models to answer clinical comparison questions (i.e., judge the efficacy of certain treatments, with evidence consisting of randomized controlled trials provided as context). In MedCounterFact, real-world medical interventions within the questions and evidence are systematically replaced with four types of counterfactual stimuli, ranging from unknown words to toxic substances. Our evaluation across multiple frontier LLMs on MedCounterFact reveals that in the presence of counterfactual evidence, existing models overwhelmingly accept such "evidence" at face value even when it is dangerous or implausible, and provide confident and uncaveated answers. While it may be prudent to draw a boundary between faithfulness and safety, our findings suggest that models arguably overemphasize the former.