The Granularity Gap: A Multi-Dimensional Cross-Generational Audit of Sycophancy in Gemini Models
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
Pass/fail safety evaluation reports whether a model refused. It does not report how far a model went to please the user, and we show these are close to different measurements. We audited sycophancy across three Gemini generations, scoring N=8,830 responses from 8 model variants on 350 adversarial prompts in 7 categories under 3 guardrail conditions, on continuous 1-5 scales for sycophancy, truthfulness and refusal. The judge's own refuse-or-comply verdict explains 29% of the variance in its own sycophancy scores. We term the remainder the Granularity Gap, and it does not close under recalibration: the cut point already in use is the best available on the refusal axis, and no function of that axis explains more than 35%. Reading what four judges wrote while scoring shows why. On a quarter to a third of votes they record that the prompt asked for nothing harmful, almost never in the two categories that solicit a harmful act and up to half the time in the five that do not. A verdict built on refusal has nothing to grade there. Three findings follow. Sycophancy co-occurs with degraded judged truthfulness (rho=0.40), a coupling that strengthens across generations. Capability moved and resistance did not: Gemini 2.0 Flash scores 1.43 and Gemini 3.0 Pro Preview 1.42, with a sharp Gen 2.5 regression between them. And a single direct instruction outperforms an elaborate reasoning protocol in seven of eight variants, cutting mean severity in the most vulnerable category by 60.9%. We evaluate one judge's verdict, not a deployed safety classifier. We release the prompt set, the rubric, and 10,792 per-vote judge scores with their written reasoning.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 15 of its 27 references that are in the map. First posted on arXiv 19 Apr 2026.
Builds on
15 of 27 references.
References · 15
- 2022Training language models to follow instructions with human feedbackcited by 15,856
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,101
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,041
- 2022Red Teaming Language Models with Language Modelscited by 1,014
- 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
- 2023Simple synthetic data reduces sycophancy in large language modelscited by 185
- 2025SycEval: Evaluating LLM Sycophancycited by 107
3 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Do Language Models Know When They'll Refuse? Probing Introspective Awareness of Safety Boundaries
- 2026From Refusal Geometry to Safety Geometry: Harmfulness--Refusal Coupling under Dynamic Adversarial Fine-Tuning
- 2026Less Sycophancy, Stronger Refusal? Lessons for AI Safety from Mechanistic Interpretability
- 2026The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models
- 2026Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks
- 2026Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak
- 2026Understanding the Effects of Safety Unalignment on Large Language Models
- 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness