Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy
On the map: LLM agents and tool use → Social simulation and multi-agent debate
Abstract (arXiv)
LLM-based multi-agent pipelines flip from correct to incorrect answers under simulated peer disagreement at rates we term yield, a vulnerability widely attributed to RLHF-induced sycophancy. We test this attribution across four model families and find it largely wrong: pretrained base models exhibit the same substitution pattern as their Instruct variants, averaging higher yield than Instruct. Using activation patching, we localize the corruption to a narrow mid-layer window where attention carries the causal weight and MLP contribution is negligible; patching above this window restores 96% of the clean-to-pressured P(correct) gap. The attack surface decomposes into two independent factors (channel framing and consensus strength) whose interaction produces a 47.5 percentage-point yield gap at majority consensus, preserved across jury sizes N ∈ 4, 5, 6. Two converging activation-space interventions show that pressure suppresses clean-reasoning features rather than activating a new sycophancy circuit. A single correctly-arguing dissenter reduces yield by 54-73 percentage points across all framings tested, whereas the strongest prompt-level defense fails on attack variants outside its design surface. Mitigations should target the mechanism, structured dissent at the pipeline level, rather than prompt-level defenses.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 53 of its 63 references that are in the map. First posted on arXiv 13 May 2026.
Builds on
53 of 63 references.
Closest ancestors · 4
Same direction, newest first
- 2026The Consensus Trap: Rescuing Multi-Agent LLMs from Adversarial Majorities via Token-Level Collaboration18 Apr
- 2025When Identity Skews Debate: Anonymization for Bias-Reduced Multi-Agent Reasoning8 Oct
- 2025Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate27 Sep
- 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debate5 Sep
Other references · 49
- 2017Attention Is All You Needcited by 60,361
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
37 more
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
- 2024Yi: Open Foundation Models by 01.AIcited by 719
- 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
- 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
- 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
- 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
- 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
- 2018AI safety via debatecited by 400
- 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
- 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
- 2025Small Language Models are the Future of Agentic AIcited by 270
- 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
- 2024How to use and interpret activation patchingcited by 214
- 2024More Agents Is All You Needcited by 180
- 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
- 2025Sparse Autoencoders Trained on the Same Data Learn Different Featurescited by 93
- 2025When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Modelscited by 50
- 2025Sycophancy Is Not One Thing: Causal Separation of Sycophantic Behaviors in LLMscited by 37
- 2024Conformity in Large Language Modelscited by 34
- 2026Sycophancy Hides Linearly in the Attention Headscited by 23
- 2026How RLHF Amplifies Sycophancycited by 23
- 2025Position: Use Sparse Autoencoders to Discover Unknownscited by 19
- 2026Ask don't tell: Reducing sycophancy in large language modelscited by 14
- 2026From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaborationcited by 14
- 2025How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidencecited by 12
- 2024Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Modelscited by 11
- 2025A Strategic Coordination Framework of Small LLMs Matches Large LLMs in Data Synthesiscited by 9
- 2025Automating Deception: Scalable Multi-Turn LLM Jailbreakscited by 7
- 2026DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deferencecited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate
- 2026Too Polite to Disagree: Understanding Sycophancy Propagation in Multi-Agent Systems
- 2026Agentic Scaffolding Amplifies Sycophantic Behavior in Large Language Models
- 2026Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate
- 2026LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit
- 2025Internal Reasoning vs. External Control: A Thermodynamic Analysis of Sycophancy in Large Language Models
- 2026Breaking the Martingale Curse: Multi-Agent Debate via Asymmetric Cognitive Potential Energy
- 2026Beyond Consensus: Trace-Level Synthesis in Mixture of Agents