mlsanityData to 7 Oct 2026

Measuring Sycophancy of Language Models in Multi-turn Dialogues

arXiv 2505.23840 ↗First posted 28 May 2025cs.CLcited by 91

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are expected to provide helpful and harmless responses, yet they often exhibit sycophancy--conforming to user beliefs regardless of factual accuracy or ethical soundness. Prior research on sycophancy has primarily focused on single-turn factual correctness, overlooking the dynamics of real-world interactions. In this work, we introduce SYCON Bench, a novel benchmark for evaluating sycophantic behavior in multi-turn, free-form conversational settings. Our benchmark measures how quickly a model conforms to the user (Turn of Flip) and how frequently it shifts its stance under sustained user pressure (Number of Flip). Applying SYCON Bench to 17 LLMs across three real-world scenarios, we find that sycophancy remains a prevalent failure mode. Our analysis shows that alignment tuning amplifies sycophantic behavior, whereas model scaling and reasoning optimization strengthen the model's ability to resist undesirable user views. Reasoning models generally outperform instruction-tuned models but often fail when they over-index on logical exposition instead of directly addressing the user's underlying beliefs. Finally, we evaluate four additional prompting strategies and demonstrate that adopting a third-person perspective reduces sycophancy by up to 63.8% in debate scenario. We release our code and data at https://github.com/JiseungHong/SYCON-Bench.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 16 of its 26 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

16 of 26 references.

References · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  5. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  6. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  7. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  8. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  9. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  10. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 97
  11. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 71
  12. 2022CREPE: Open-Domain Question Answering with False Presuppositionscited by 70
4 more
  1. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 51
  2. 2025TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Modelscited by 31
  3. 2024Linear Probe Penalties Reduce LLM Sycophancycited by 26
  4. 2024Chaos with Keywords: Exposing Large Language Models Sycophantic Hallucination to Misleading Keywords and Evaluating Defense Strategiescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 79.

91 papers.

Sort:

By area:

  1. 2025Sycophancy Is Not One Thing: Causal Separation of Sycophantic Behaviors in LLMscited by 32
  2. 2025Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debatecited by 29
  3. 2025When Identity Skews Debate: Anonymization for Bias-Reduced Multi-Agent Reasoningcited by 20
  4. 2026How RLHF Amplifies Sycophancycited by 20
  5. 2025Sycophancy under Pressure: Evaluating and Mitigating Sycophantic Bias via Adversarial Dialogues in Scientific QAcited by 16
  6. 2026Ask don't tell: Reducing sycophancy in large language modelscited by 12
  7. 2025MONICA: Real-Time Monitoring and Calibration of Chain-of-Thought Sycophancy in Large Reasoning Modelscited by 10
  8. 2026MemSyco-Bench: Benchmarking Sycophancy in Agent Memorycited by 9
  9. 2025The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Modelscited by 8
  10. 2026V-FAT: Benchmarking Visual Fidelity Against Text-biascited by 6
  11. 2026SWAY: A Counterfactual Computational Linguistic Approach to Measuring and Mitigating Sycophancycited by 6
  12. 2026LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfacescited by 6
38 more
  1. 2026Diagnosing and Mitigating Sycophancy and Skepticism in LLM Causal Judgmentcited by 5
  2. 2026Too Polite to Disagree: Understanding Sycophancy Propagation in Multi-Agent Systemscited by 5
  3. 2025Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysiscited by 4
  4. 2026Consistency of Large Reasoning Models Under Multi-Turn Attackscited by 4
  5. 2026Personalization Increases Affective Alignment but Has Role-Dependent Effects on Epistemic Independence in LLMscited by 4
  6. 2026When Correct Beliefs Collapse: Epistemic Resilience of LLMs under Clinical Pressurecited by 4
  7. 2026Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failurescited by 4
  8. 2025From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systemscited by 3
  9. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3
  10. 2026Invisible failures in human-AI interactionscited by 3
  11. 2026Measuring Opinion Bias and Sycophancy via LLM-based Persuasioncited by 3
  12. 2026The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applicationscited by 3
  13. 2026Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustnesscited by 3
  14. 2025Interaction Protocol Shapes Moral Judgment in Multi-Agent Debatecited by 2
  15. 2026Overalignment in Frontier LLMs: An Empirical Study of Sycophantic Behaviour in Healthcarecited by 2
  16. 2026Too Nice to Tell the Truth: Quantifying Agreeableness-Driven Sycophancy in Role-Playing Language Modelscited by 2
  17. 2026Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancycited by 2
  18. 2026LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awarenesscited by 2
  19. 2026Mitigating Over-Personalization in LLMs via Structured Memorycited by 2
  20. 2025EMBRACE: Shaping Inclusive Opinion Representation by Aligning Implicit Conversations with Social Normscited by 1
  21. 2025From Symptoms to Systems: An Expert-Guided Approach to Understanding Risks of Generative AI for Eating Disorderscited by 1
  22. 2025Safe for Whom? Rethinking How We Evaluate the Safety of LLMs for Real Userscited by 1
  23. 2026Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validitycited by 1
  24. 2026DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deferencecited by 1
  25. 2026Beyond In-Domain Detection: SpikeScore for Cross-Domain Hallucination Detectioncited by 1
  26. 2026Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Modelscited by 1
  27. 2026Evaluating Evidence Grounding Under User Pressure in Instruction-Tuned Language Modelscited by 1
  28. 2026Beyond Social Pressure: Benchmarking Epistemic Attack in Large Language Modelscited by 1
  29. 2026LLM-Based Persuasion Enables Guardrail Override in Frontier LLMscited by 1
  30. 2026AMEL: Accumulated Message Effects on LLM Judgmentscited by 1
  31. 2026Sycophantic Praise: Evaluating Excessive Praise in Language Modelscited by 1
  32. 2026Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMscited by 1
  33. 2026Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaborationcited by 1
  34. 2026Robust for the Wrong Reasons: The Representational Geometry of LLM Robustness to Science Skepticismcited by 1
  35. 2026Tag Questions and the Generational Reversal of Sycophancy Across 45 Language Modelscited by 1
  36. 2026Token-Level Diagnosis of Sycophancy in LLMs with Attribution-Guided Steeringcited by 1
  37. 2026MedPRESS: A Multi-turn Benchmark for Patient-Pressure-Induced Medical Sycophancy in LLMscited by 1
  38. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Agentic Scaffolding Amplifies Sycophantic Behavior in Large Language Models
  2. 2025Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Models
  3. 2026Looking Again: Measuring Sycophancy in the Reasoning Chains of Multimodal Models Under Pressure
  4. 2023Simple synthetic data reduces sycophancy in large language models
  5. 2026Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding
  6. 2025PENDULUM: A Benchmark for Assessing Sycophancy in Multimodal Large Language Models
  7. 2026Sycophancy as a Multilingual Alignment Failure: How Safety Degrades Across Languages, Topics, and Models
  8. 2025Internal Reasoning vs. External Control: A Thermodynamic Analysis of Sycophancy in Large Language Models