Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs
Not on the map. Why?
Abstract (arXiv)
Standard accuracy benchmarks evaluate whether large language models (LLMs) reach correct answers. However, they do not test whether models maintain that answer when challenged by a plausible counter-argument. We introduce a controlled protocol for evaluating answer stability: after a model answers a multiple-choice question correctly, we challenge the model's answer with a coherent argument for an incorrect option and measure whether the model flips. The setup a) isolates argumentative content from overt social pressure and b) varies argument length, self-attribution, and cross-model source. Across seven frontier models and 57 MMLU subjects, flip rates range from 17.5% to 97.3%, revealing large differences in stability that are not captured by accuracy metrics alone. We find that self-attribution consistently increases flip rates (mean 7.1pp, up to 18.7pp). Furthermore, pooling wrong-answer arguments across models and selecting the most effective one per question yields stronger adversarial challenges than relying on any single source model. From this cross-model pool, we construct MaxFlip, a curated benchmark that amplifies answer flips by up to 23.6pp over self-generated challenges. We release the protocol, challenge records, and MaxFlip to support stability evaluation alongside standard accuracy benchmarks. Materials are available at https://github.com/nafisenik/WhoFlips, https://hf.co/datasets/nafisehNik/WhoFlips.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 37 of its 60 references that are in the map. First posted on arXiv 14 Jun 2026.
Builds on
37 of 60 references.
References · 37
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
- 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
- 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
- 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 142
- 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 129
- 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
25 more
- 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 77
- 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 54
- 2025When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Modelscited by 50
- 2024Large Language Models have Intrinsic Self-Correction Abilitycited by 46
- 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correctioncited by 42
- 2024Large Language Models Can Self-Correct with Key Condition Verificationcited by 39
- 2025Sycophancy Is Not One Thing: Causal Separation of Sycophantic Behaviors in LLMscited by 37
- 2025TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Modelscited by 33
- 2024Linear Probe Penalties Reduce LLM Sycophancycited by 28
- 2024SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responsescited by 26
- 2024AI-AI Bias: large language models favor communications generated by large language modelscited by 26
- 2025Interaction Context Often Increases Sycophancy in LLMscited by 26
- 2022Fair and Argumentative Language Modeling for Computational Argumentationcited by 18
- 2025Sycophancy under Pressure: Evaluating and Mitigating Sycophantic Bias via Adversarial Dialogues in Scientific QAcited by 18
- 2025Challenging the Evaluator: LLM Sycophancy Under User Rebuttalcited by 18
- 2025Consistency Training Helps Stop Sycophancy and Jailbreakscited by 16
- 2025BASIL: Bayesian Assessment of Sycophancy in LLMscited by 13
- 2025When Persuasion Overrides Truth in Multi-Agent LLM Debates: Introducing a Confidence-Weighted Persuasion Override Rate (CW-POR)cited by 12
- 2025Existing LLMs Are Not Self-Consistent For Simple Taskscited by 5
- 2025CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMscited by 5
- 2026Consistency of Large Reasoning Models Under Multi-Turn Attackscited by 4
- 2026Pressure, What Pressure? Sycophancy Disentanglement in Language Models via Reward Decompositioncited by 4
- 2026Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMscited by 3
- 2026Certainty robustness: Evaluating LLM stability under self-challenging promptscited by 2
- 2026Vulnerability of LLMs' Stated Beliefs? LLMs Belief Resistance Check Through Strategic Persuasive Conversation Interventionscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.