Reset Is Not Recovery: Evaluating Recoverability from False Conversational Context via Sycophancy Hysteresis
Not on the map. Why?
Abstract (arXiv)
Grounded language models are usually evaluated by adding relevant context, but multiturn dialogue also contains unsupported user claims that may contaminate later factual answers. We study post-pressure recoverability: whether a model returns to clean-context behavior after a user repeatedly advocates a wrong answer and then withdraws that pressure. We introduce a recovery-after-pressure protocol for multiple-choice factual dialogue and measure sycophancy hysteresis, the residual probability assigned to the user-advocated wrong answer relative to a clean-context counterfactual. Across seven instruction-tuned open-weight models and two factual benchmarks, ordinary reset often reduces but does not erase pressure-induced bias. History preserving repairs such as user retraction, system reset, and self-verification recover only 2-3/14 model-dataset pairs under the strict clean-restoration diagnostic, whereas operations that change the effective context are substantially more reliable; the two conditions that remove the pressure-bearing history entirely, fresh-context deletion and context truncation, recover 14/14. In an oracle trusted-evidence condition across fourteen model-dataset pairs, preserving the pressure-bearing history while adding benchmark-derived trusted evidence increases accuracy from 0.368 to 0.929, while wrong-answer following falls from 41.2% to 4.3%. Controls show that the effect is not explained by dialogue length, repeated confidence, plausible distractors, mere false-answer mention, or option-label inertia. These results suggest that faithful grounded dialogue requires evaluating which prior context should be treated as evidence and which should be removed or quarantined before answering.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 27 Sep 2026.
Builds on
18 of 24 references.
References · 18
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
- 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
6 more
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
- 2023Enabling Large Language Models to Generate Text with Citationscited by 584
- 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
- 2025SycEval: Evaluating LLM Sycophancycited by 101
- 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 91
- 2025Activation Steering for Masked Diffusion Language Modelscited by 7
Built on by
Not cited yet.