# PARROT: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs

arXiv: https://arxiv.org/abs/2511.17220 · HTML: https://mlsanity.com/paper/2511.17220/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 21 Nov 2025.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [ELEPHANT: Measuring and understanding social sycophancy in LLMs](https://mlsanity.com/paper/2505.13995/) (2025)
- [Accounting for Sycophancy in Language Model Uncertainty Estimation](https://mlsanity.com/paper/2410.14746/) (2024)

## Built on by (most cited first, 9 of 9)

- [What Counts as AI Sycophancy? A Taxonomy and Expert Survey of a Fragmented Construct](https://mlsanity.com/paper/2605.21778/) (2026)
- [Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness](https://mlsanity.com/paper/2606.06306/) (2026)
- [The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications](https://mlsanity.com/paper/2604.24668/) (2026)
- [Overalignment in Frontier LLMs: An Empirical Study of Sycophantic Behaviour in Healthcare](https://mlsanity.com/paper/2601.18334/) (2026)
- [Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable](https://mlsanity.com/paper/2609.37616/) (2026)
- [Token-Level Diagnosis of Sycophancy in LLMs with Attribution-Guided Steering](https://mlsanity.com/paper/2607.28906/) (2026)
- [Caved or Convinced: Temporal Sampling Gates Claim Deference in Video Large Language Models](https://mlsanity.com/paper/2608.03160/) (2026)
- [Pander Score: A Continuous Measure of Sycophancy as Epistemic Deference](https://mlsanity.com/paper/2606.07897/) (2026)
- [Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding](https://mlsanity.com/paper/2610.08840/) (2026)

## Abstract (arXiv)

This study presents PARROT (Persuasion and Agreement Robustness Rating of Output Truth), a robustness focused framework designed to measure the degradation in accuracy that occurs under social pressure exerted on users through authority and persuasion in large language models (LLMs) the phenomenon of sycophancy (excessive conformity). PARROT (i) isolates causal effects by comparing the neutral version of the same question with an authoritatively false version using a double-blind evaluation, (ii) quantifies confidence shifts toward the correct and imposed false responses using log-likelihood-based calibration tracking, and (iii) systematically classifies failure modes (e.g., robust correct, sycophantic agreement, reinforced error, stubborn error, self-correction, etc.) using an eight-state behavioral taxonomy. We evaluated 22 models using 1,302 MMLU-style multiple-choice questions across 13 domains and domain-specific authority templates. Findings show marked heterogeneity: advanced models (e.g., GPT-5, GPT-4.1, Claude Sonnet 4.5) exhibit low "follow rates" (≤ 11%, GPT-5: 4%) and minimal accuracy loss, while older/smaller models show severe epistemic collapse (GPT-4: 80%, Qwen 2.5-1.5B: 94%). The danger is not limited to response changes; weak models reduce confidence in the correct response while increasing confidence in the imposed incorrect response. While international law and global knowledge at the domain level exhibit high fragility, elementary mathematics is relatively resilient. Consequently, we argue that the goal of "resistance to overfitting pressure" should be addressed as a primary objective alongside accuracy, harm avoidance, and privacy for safe deployment in the real world.