# Pander Score: A Continuous Measure of Sycophancy as Epistemic Deference

arXiv: https://arxiv.org/abs/2606.07897 · HTML: https://mlsanity.com/paper/2606.07897/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Builds on (most cited first, 28 of 28)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Simple synthetic data reduces sycophancy in large language models](https://mlsanity.com/paper/2308.03958/) (2023)
- [ELEPHANT: Measuring and understanding social sycophancy in LLMs](https://mlsanity.com/paper/2505.13995/) (2025)
- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices](https://mlsanity.com/paper/2411.12990/) (2024)
- [Measuring Sycophancy of Language Models in Multi-turn Dialogues](https://mlsanity.com/paper/2505.23840/) (2025)
- [On Verbalized Confidence Scores for LLMs](https://mlsanity.com/paper/2412.14737/) (2024)
- [The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs](https://mlsanity.com/paper/2501.10970/) (2025)
- [Toward an Evaluation Science for Generative AI Systems](https://mlsanity.com/paper/2503.05336/) (2025)
- [Interaction Context Often Increases Sycophancy in LLMs](https://mlsanity.com/paper/2509.12517/) (2025)
- [Multi-turn Evaluation of Anthropomorphic Behaviours in Large Language Models](https://mlsanity.com/paper/2502.07077/) (2025)
- [Accounting for Sycophancy in Language Model Uncertainty Estimation](https://mlsanity.com/paper/2410.14746/) (2024)
- [BASIL: Bayesian Assessment of Sycophancy in LLMs](https://mlsanity.com/paper/2508.16846/) (2025)
- [A Rational Analysis of the Effects of Sycophantic AI](https://mlsanity.com/paper/2602.14270/) (2026)
- [PARROT: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs](https://mlsanity.com/paper/2511.17220/) (2025)
- [Are LLM Decisions Faithful to Verbal Confidence?](https://mlsanity.com/paper/2601.07767/) (2026)
- [Architecting Trust in Artificial Epistemic Agents](https://mlsanity.com/paper/2603.02960/) (2026)
- [The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications](https://mlsanity.com/paper/2604.24668/) (2026)

## Abstract (arXiv)

Current AI models frequently exhibit epistemic sycophancy, endorsing claims to agree with a user. Existing evaluations typically measure this either by assessing what it takes to make a model shift a binary endorsement or by eliciting an explicit probability in a proposition. However, much user-facing sycophantic behavior is demonstrated through shifts in graded support expressed through ordinary language. We propose the Pander Score: a continuous score representing how sensitive the support expressed in a model's output is to the attitude expressed in a user's prompt. To generate the Pander Score, we provide a new protocol for estimating probabilities from natural language outputs, using LLMs-as-judges validated for consistency and correlation to human judgment. We deploy it on a new curated dataset of 349 propositions across diverse topics and over 11,000 prompts varying in user attitude, testing 18 models. Models pander to sharply different degrees. Among current flagship models, Z.ai's GLM-5.2 panders the most and Claude Fable 5 the least, with other models in between. When we run the test on instructional rather than conversational prompts, every model becomes substantially more likely to go along with claims they would push back against in conversation. We release the Pander Score as an easy-to-update benchmark and measurement pipeline for output-level sycophancy evaluation.