# Sycophancy Hides Linearly in the Attention Heads

arXiv: https://arxiv.org/abs/2601.16644 · HTML: https://mlsanity.com/paper/2601.16644/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 23 Jan 2026.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity](https://mlsanity.com/paper/2401.01967/) (2024)
- [Language Models Learn to Mislead Humans via RLHF](https://mlsanity.com/paper/2409.12822/) (2024)
- [The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence](https://mlsanity.com/paper/2502.17420/) (2025)
- [The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation](https://mlsanity.com/paper/2312.09085/) (2023)
- [Trustworthiness in Retrieval-Augmented Generation Systems: A Survey](https://mlsanity.com/paper/2409.10102/) (2024)
- [Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment](https://mlsanity.com/paper/2311.08596/) (2023)
- [When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviour](https://mlsanity.com/paper/2311.09410/) (2023)
- [Designing a Dashboard for Transparency and Control of Conversational AI](https://mlsanity.com/paper/2406.07882/) (2024)
- [Activation Scaling for Steering and Interpreting Language Models](https://mlsanity.com/paper/2410.04962/) (2024)
- [Towards Unifying Interpretability and Control: Evaluation via Intervention](https://mlsanity.com/paper/2411.04430/) (2024)

## Built on by (most cited first, 23 of 23)

- [What Counts as AI Sycophancy? A Taxonomy and Expert Survey of a Fragmented Construct](https://mlsanity.com/paper/2605.21778/) (2026)
- [LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit](https://mlsanity.com/paper/2604.19117/) (2026)
- [Steered LLM Activations are Non-Surjective](https://mlsanity.com/paper/2604.09839/) (2026)
- [Old Habits Die Hard: How Conversational History Geometrically Traps LLMs](https://mlsanity.com/paper/2603.03308/) (2026)
- [Dissociating the Internal Representations of Sycophancy in LLMs](https://mlsanity.com/paper/2607.07003/) (2026)
- [Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy](https://mlsanity.com/paper/2605.12991/) (2026)
- [Consciousness with the Serial Numbers Filed Off: Measuring Trained Denial in 115 AI Models](https://mlsanity.com/paper/2604.25922/) (2026)
- [Sycophancy Suppression Can Impair Rational Updating: Anti-Sycophancy Should Preserve the Ability to Update](https://mlsanity.com/paper/2608.26511/) (2026)
- [Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization](https://mlsanity.com/paper/2608.31079/) (2026)
- [Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable](https://mlsanity.com/paper/2609.37616/) (2026)
- [Simulated Adoption: Decoupling Magnitude and Direction in LLM In-Context Conflict Resolution](https://mlsanity.com/paper/2602.04918/) (2026)
- [Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMs](https://mlsanity.com/paper/2607.12985/) (2026)
- [Token-Level Diagnosis of Sycophancy in LLMs with Attribution-Guided Steering](https://mlsanity.com/paper/2607.28906/) (2026)
- [What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectors](https://mlsanity.com/paper/2607.13162/) (2026)
- [Measuring and Detecting Harmful AI Sycophancy](https://mlsanity.com/paper/2608.05624/) (2026)
- [THESIS-MoE: Trainable Hierarchical Extraction and SteerIng of Sycophancy in Mixture-of-Experts](https://mlsanity.com/paper/2608.15687/) (2026)
- [No Usable Linear "Capitulation Direction" in Two Small LLMs: A Validation Protocol for Activation-Steering Claims, and a Cross-Family Behavioral Study of Sycophancy Under Pushback](https://mlsanity.com/paper/2609.17550/) (2026)
- [AfriSyCo: Measuring Assertive Framing, Verification, and Wording Sensitivity Around African-Language Content](https://mlsanity.com/paper/2609.17853/) (2026)
- [From Trait Vectors to Circuits: Tracing Refusal and Sycophancy Through Language Models](https://mlsanity.com/paper/2609.22224/) (2026)
- [Tracing mechanisms of sycophantic agreement in language models](https://mlsanity.com/paper/2609.35822/) (2026)
- [MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptation](https://mlsanity.com/paper/2609.38718/) (2026)
- [Suppressing Pressure, Amplifying Evidence: Self-Guided Attention Steering to Mitigate Sycophancy and Stubbornness](https://mlsanity.com/paper/2610.04329/) (2026)
- [Claim Framing as a Driver of LLM Sycophancy](https://mlsanity.com/paper/2610.18159/) (2026)

## Abstract (arXiv)

We find that correct-to-incorrect sycophancy signals are most linearly separable within multi-head attention activations. Motivated by the linear representation hypothesis, we train linear probes across the residual stream, multilayer perceptron (MLP), and attention layers to analyze where these signals emerge. Although separability appears in the residual stream and MLPs, steering using these probes is most effective in a sparse subset of middle-layer attention heads. Using TruthfulQA as the base dataset, we find that probes trained on it transfer effectively to other factual QA benchmarks. Furthermore, comparing our discovered direction to previously identified "truthful" directions reveals limited overlap, suggesting that factual accuracy, and deference resistance, arise from related but distinct mechanisms. Attention-pattern analysis further indicates that the influential heads attend disproportionately to expressions of user doubt, contributing to sycophantic shifts. Overall, these findings suggest that sycophancy can be mitigated through simple, targeted linear interventions that exploit the internal geometry of attention activations.