# Measuring Sycophancy of Language Models in Multi-turn Dialogues

arXiv: https://arxiv.org/abs/2505.23840 · HTML: https://mlsanity.com/paper/2505.23840/

## Key facts

- Cited by 104 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 16 of 16)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [TravelPlanner: A Benchmark for Real-World Planning with Language Agents](https://mlsanity.com/paper/2402.01622/) (2024)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [Sycophancy in Large Language Models: Causes and Mitigations](https://mlsanity.com/paper/2411.15287/) (2024)
- [Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment](https://mlsanity.com/paper/2311.08596/) (2023)
- [CREPE: Open-Domain Question Answering with False Presuppositions](https://mlsanity.com/paper/2211.17257/) (2022)
- [From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning](https://mlsanity.com/paper/2409.01658/) (2024)
- [TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Models](https://mlsanity.com/paper/2503.11656/) (2025)
- [Linear Probe Penalties Reduce LLM Sycophancy](https://mlsanity.com/paper/2412.00967/) (2024)
- [Chaos with Keywords: Exposing Large Language Models Sycophantic Hallucination to Misleading Keywords and Evaluating Defense Strategies](https://mlsanity.com/paper/2406.03827/) (2024)

## Built on by (most cited first, 30 of 104)

- [Sycophancy Is Not One Thing: Causal Separation of Sycophantic Behaviors in LLMs](https://mlsanity.com/paper/2509.21305/) (2025)
- [Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate](https://mlsanity.com/paper/2509.23055/) (2025)
- [How RLHF Amplifies Sycophancy](https://mlsanity.com/paper/2602.01002/) (2026)
- [When Identity Skews Debate: Anonymization for Bias-Reduced Multi-Agent Reasoning](https://mlsanity.com/paper/2510.07517/) (2025)
- [Sycophancy under Pressure: Evaluating and Mitigating Sycophantic Bias via Adversarial Dialogues in Scientific QA](https://mlsanity.com/paper/2508.13743/) (2025)
- [Ask don't tell: Reducing sycophancy in large language models](https://mlsanity.com/paper/2602.23971/) (2026)
- [MONICA: Real-Time Monitoring and Calibration of Chain-of-Thought Sycophancy in Large Reasoning Models](https://mlsanity.com/paper/2511.06419/) (2025)
- [MemSyco-Bench: Benchmarking Sycophancy in Agent Memory](https://mlsanity.com/paper/2607.01071/) (2026)
- [The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models](https://mlsanity.com/paper/2509.10970/) (2025)
- [SWAY: A Counterfactual Computational Linguistic Approach to Measuring and Mitigating Sycophancy](https://mlsanity.com/paper/2604.02423/) (2026)
- [Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures](https://mlsanity.com/paper/2609.29429/) (2026)
- [V-FAT: Benchmarking Visual Fidelity Against Text-bias](https://mlsanity.com/paper/2601.04897/) (2026)
- [Too Polite to Disagree: Understanding Sycophancy Propagation in Multi-Agent Systems](https://mlsanity.com/paper/2604.02668/) (2026)
- [LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces](https://mlsanity.com/paper/2604.06188/) (2026)
- [When Correct Beliefs Collapse: Epistemic Resilience of LLMs under Clinical Pressure](https://mlsanity.com/paper/2605.23932/) (2026)
- [Diagnosing and Mitigating Sycophancy and Skepticism in LLM Causal Judgment](https://mlsanity.com/paper/2601.08258/) (2026)
- [Personalization Increases Affective Alignment but Has Role-Dependent Effects on Epistemic Independence in LLMs](https://mlsanity.com/paper/2603.00024/) (2026)
- [Automated Researchers Can Mitigate Well-characterized Alignment Failures](https://mlsanity.com/paper/2608.28945/) (2026)
- [Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis](https://mlsanity.com/paper/2510.25356/) (2025)
- [Consistency of Large Reasoning Models Under Multi-Turn Attacks](https://mlsanity.com/paper/2602.13093/) (2026)
- [LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness](https://mlsanity.com/paper/2605.26438/) (2026)
- [Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness](https://mlsanity.com/paper/2606.06306/) (2026)
- [Interaction Protocol Shapes Moral Judgment in Multi-Agent Debate](https://mlsanity.com/paper/2510.10002/) (2025)
- [From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems](https://mlsanity.com/paper/2511.10871/) (2025)
- [One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models](https://mlsanity.com/paper/2603.03291/) (2026)
- [Invisible failures in human-AI interactions](https://mlsanity.com/paper/2603.15423/) (2026)
- [Too Nice to Tell the Truth: Quantifying Agreeableness-Driven Sycophancy in Role-Playing Language Models](https://mlsanity.com/paper/2604.10733/) (2026)
- [Measuring Opinion Bias and Sycophancy via LLM-based Persuasion](https://mlsanity.com/paper/2604.21564/) (2026)
- [The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications](https://mlsanity.com/paper/2604.24668/) (2026)
- [Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy](https://mlsanity.com/paper/2605.12991/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are expected to provide helpful and harmless responses, yet they often exhibit sycophancy--conforming to user beliefs regardless of factual accuracy or ethical soundness. Prior research on sycophancy has primarily focused on single-turn factual correctness, overlooking the dynamics of real-world interactions. In this work, we introduce SYCON Bench, a novel benchmark for evaluating sycophantic behavior in multi-turn, free-form conversational settings. Our benchmark measures how quickly a model conforms to the user (Turn of Flip) and how frequently it shifts its stance under sustained user pressure (Number of Flip). Applying SYCON Bench to 17 LLMs across three real-world scenarios, we find that sycophancy remains a prevalent failure mode. Our analysis shows that alignment tuning amplifies sycophantic behavior, whereas model scaling and reasoning optimization strengthen the model's ability to resist undesirable user views. Reasoning models generally outperform instruction-tuned models but often fail when they over-index on logical exposition instead of directly addressing the user's underlying beliefs. Finally, we evaluate four additional prompting strategies and demonstrate that adopting a third-person perspective reduces sycophancy by up to 63.8% in debate scenario. We release our code and data at https://github.com/JiseungHong/SYCON-Bench.