# Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs

arXiv: https://arxiv.org/abs/2606.16011 · HTML: https://mlsanity.com/paper/2606.16011/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 60 references that are in the map.
- First posted on arXiv 14 Jun 2026.

## Builds on (most cited first, 30 of 37)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://mlsanity.com/paper/2406.01297/) (2024)
- [On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks](https://mlsanity.com/paper/2402.08115/) (2024)
- [ELEPHANT: Measuring and understanding social sycophancy in LLMs](https://mlsanity.com/paper/2505.13995/) (2025)
- [Measuring Sycophancy of Language Models in Multi-turn Dialogues](https://mlsanity.com/paper/2505.23840/) (2025)
- [Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment](https://mlsanity.com/paper/2311.08596/) (2023)
- [From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning](https://mlsanity.com/paper/2409.01658/) (2024)
- [When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Models](https://mlsanity.com/paper/2508.02087/) (2025)
- [Large Language Models have Intrinsic Self-Correction Ability](https://mlsanity.com/paper/2406.15673/) (2024)
- [Understanding the Dark Side of LLMs' Intrinsic Self-Correction](https://mlsanity.com/paper/2412.14959/) (2024)
- [Large Language Models Can Self-Correct with Key Condition Verification](https://mlsanity.com/paper/2405.14092/) (2024)
- [Sycophancy Is Not One Thing: Causal Separation of Sycophantic Behaviors in LLMs](https://mlsanity.com/paper/2509.21305/) (2025)
- [TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Models](https://mlsanity.com/paper/2503.11656/) (2025)
- [Linear Probe Penalties Reduce LLM Sycophancy](https://mlsanity.com/paper/2412.00967/) (2024)
- [SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responses](https://mlsanity.com/paper/2404.04298/) (2024)
- [AI-AI Bias: large language models favor communications generated by large language models](https://mlsanity.com/paper/2407.12856/) (2024)
- [Interaction Context Often Increases Sycophancy in LLMs](https://mlsanity.com/paper/2509.12517/) (2025)
- [Fair and Argumentative Language Modeling for Computational Argumentation](https://mlsanity.com/paper/2204.04026/) (2022)
- [Sycophancy under Pressure: Evaluating and Mitigating Sycophantic Bias via Adversarial Dialogues in Scientific QA](https://mlsanity.com/paper/2508.13743/) (2025)
- [Challenging the Evaluator: LLM Sycophancy Under User Rebuttal](https://mlsanity.com/paper/2509.16533/) (2025)
- [Consistency Training Helps Stop Sycophancy and Jailbreaks](https://mlsanity.com/paper/2510.27062/) (2025)
- [BASIL: Bayesian Assessment of Sycophancy in LLMs](https://mlsanity.com/paper/2508.16846/) (2025)
- [When Persuasion Overrides Truth in Multi-Agent LLM Debates: Introducing a Confidence-Weighted Persuasion Override Rate (CW-POR)](https://mlsanity.com/paper/2504.00374/) (2025)

## Built on by (most cited first, 1 of 1)

- [Nudgeability: Reasoning Models Follow Confidence Signals Without Tracking Their Own Competence](https://mlsanity.com/paper/2609.34572/) (2026)

## Abstract (arXiv)

Standard accuracy benchmarks evaluate whether large language models (LLMs) reach correct answers. However, they do not test whether models maintain that answer when challenged by a plausible counter-argument. We introduce a controlled protocol for evaluating answer stability: after a model answers a multiple-choice question correctly, we challenge the model's answer with a coherent argument for an incorrect option and measure whether the model flips. The setup a) isolates argumentative content from overt social pressure and b) varies argument length, self-attribution, and cross-model source. Across seven frontier models and 57 MMLU subjects, flip rates range from 17.5% to 97.3%, revealing large differences in stability that are not captured by accuracy metrics alone. We find that self-attribution consistently increases flip rates (mean 7.1pp, up to 18.7pp). Furthermore, pooling wrong-answer arguments across models and selecting the most effective one per question yields stronger adversarial challenges than relying on any single source model. From this cross-model pool, we construct MaxFlip, a curated benchmark that amplifies answer flips by up to 23.6pp over self-generated challenges. We release the protocol, challenge records, and MaxFlip to support stability evaluation alongside standard accuracy benchmarks. Materials are available at https://github.com/nafisenik/WhoFlips, https://hf.co/datasets/nafisehNik/WhoFlips.