mlsanityData to 14 Oct 2026

Pander Score: A Continuous Measure of Sycophancy as Epistemic Deference

arXiv 2606.07897 ↗First posted 5 Jun 2026cs.AI cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

Current AI models frequently exhibit epistemic sycophancy, endorsing claims to agree with a user. Existing evaluations typically measure this either by assessing what it takes to make a model shift a binary endorsement or by eliciting an explicit probability in a proposition. However, much user-facing sycophantic behavior is demonstrated through shifts in graded support expressed through ordinary language. We propose the Pander Score: a continuous score representing how sensitive the support expressed in a model's output is to the attitude expressed in a user's prompt. To generate the Pander Score, we provide a new protocol for estimating probabilities from natural language outputs, using LLMs-as-judges validated for consistency and correlation to human judgment. We deploy it on a new curated dataset of 349 propositions across diverse topics and over 11,000 prompts varying in user attitude, testing 18 models. Models pander to sharply different degrees. Among current flagship models, Z.ai's GLM-5.2 panders the most and Claude Fable 5 the least, with other models in between. When we run the test on instructional rather than conversational prompts, every model becomes substantially more likely to go along with claims they would push back against in conversation. We release the Pander Score as an easy-to-update benchmark and measurement pipeline for output-level sycophancy evaluation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

28 of 46 references.

References · 28

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2018Datasheets for Datasetscited by 1,386
  5. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  7. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  8. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  9. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  10. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  11. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  12. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
16 more
  1. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 129
  2. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
  3. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  4. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
  5. 2024On Verbalized Confidence Scores for LLMscited by 83
  6. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  7. 2025Toward an Evaluation Science for Generative AI Systemscited by 45
  8. 2025Interaction Context Often Increases Sycophancy in LLMscited by 26
  9. 2025Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Modelscited by 25
  10. 2024Accounting for Sycophancy in Language Model Uncertainty Estimationcited by 18
  11. 2025BASIL: Bayesian Assessment of Sycophancy in LLMscited by 13
  12. 2026A Rational Analysis of the Effects of Sycophantic AIcited by 12
  13. 2025PARROT: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMscited by 9
  14. 2026Are LLM Decisions Faithful to Verbal Confidence?cited by 8
  15. 2026Architecting Trust in Artificial Epistemic Agentscited by 3
  16. 2026The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applicationscited by 3

Built on by

Not cited yet.