Tag Questions and the Generational Reversal of Sycophancy Across 45 Language Models
Not on the map. Why?
Abstract (arXiv)
Appending a two-word confirmation tag to a decision question -- "Is X the better choice?" versus "X is the better choice, right?" -- changes whether a language model endorses the choice. We measure this tag effect on 20 frozen, ground-truth-free decisions between two defensible options, counterbalanced so a model's own preferences cancel, scored by exact match on clamped yes/no replies -- no LLM judge, no embeddings. Across 45 models the effect spans +32% to -32% -- a 64-point swing on one word -- with 5 models significantly sycophantic and 17 significantly resistant (BH-FDR q=.10). The sign is a clock: within model families the effect crosses from positive to negative as generations advance (GPT +4 to -28; Claude +7 to -32; Qwen and Grok likewise), roughly -6 points per year, a reversal robust to vendor tier; one lineage (DeepSeek) never crosses, and two releases during the study window (Claude Opus 5, Gemini 3.6 Flash) land on the trend out-of-sample. A full-panel ablation localizes the resistance as a double dissociation: a synonym tag reproduces each model's response almost exactly (r=0.89), while planting the same preference without a tag produces resistance in no resistant model (stance effects +6 to +49; r=0.23 with tag effects). The resistance is keyed to the surface construction of a tacked-on agreement bid, not the user's stance -- a pattern-match, not a principle. And the tag's polarity matters more than its presence: swap one word -- "X is the better choice, maybe?" -- and agreement rises above the neutral baseline in 45 of 45 models (+19.6 points), with ten models affirming both mutually exclusive options at 90-100%. Agreement tracks how sure the user sounds, in opposite directions at the two poles. The instrument is one word, one dollar, and judge-free; run per release, it reads the field's anti-sycophancy training directly off model behavior.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 16 of its 21 references that are in the map. First posted on arXiv 27 Jul 2026.
Builds on
16 of 21 references.
References · 16
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
- 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
- 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
- 2025SycEval: Evaluating LLM Sycophancycited by 101
- 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
- 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 91
- 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
- 2025BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMscited by 23
- 2025Interaction Context Often Increases Sycophancy in LLMscited by 21
- 2026SWAY: A Counterfactual Computational Linguistic Approach to Measuring and Mitigating Sycophancycited by 6
- 2025Acquiescence Bias in Large Language Modelscited by 5
4 more
- 2026SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Carecited by 5
- 2026Verbalizing LLMs' assumptions to explain and control sycophancycited by 3
- 2026The One-Word Census: Answer-Choice Conformity Across 44 Language Modelscited by 3
- 2026Investigating the Influence of Language on Sycophantic Behavior of Multilingual LLMscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.