mlsanityData to 7 Oct 2026

How LLMs Are Persuaded: A Few Attention Heads, Rerouted

arXiv 2605.09314 ↗First posted 10 May 2026cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Large language models can answer a factual question correctly, yet switch to an incorrect answer when exposed to persuasive context. Why this happens inside the model remains unclear. In this work, we study this question through causal interventions in controlled multiple-choice tasks and identify that persuasion mainly acts through a small set of mid-layer attention heads, which we call decision heads. These heads do not compute the answer themselves. Instead, they copy the option they attend to, so changing their attention can directly change the model's answer. We find that this attention is controlled by a one-dimensional option-routing feature in the option-token representations. Persuasive text changes this feature, causing the decision heads to attend to the persuasion target instead of the correct option. We further trace the routing feature to shallower attention heads that read persuasive keywords from the input. Together, these results reveal a causal pathway: shallow heads read persuasive keywords and write the routing feature, which redirects the decision heads' attention toward the persuasion target; the decision heads then copy the selected option, leading to the wrong answer. We verify each step through intervention and find the same mechanism across four open-source model families and in a source-selection task derived from Generative Engine Optimization (GEO). Finally, based on this mechanism, we introduce a low-rank update to the decision heads' key projections to reduce their sensitivity to persuasion.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 28 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

25 of 28 references.

References · 25

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  5. 2022Mass-Editing Memory in a Transformercited by 976
  6. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  7. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  8. 2024How to use and interpret activation patchingcited by 191
  9. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  10. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  11. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  12. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
13 more
  1. 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 107
  2. 2025SycEval: Evaluating LLM Sycophancycited by 101
  3. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  4. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
  5. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 71
  6. 2023GEO: Generative Engine Optimizationcited by 53
  7. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 42
  8. 2025Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainerscited by 41
  9. 2025BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMscited by 23
  10. 2025Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMscited by 20
  11. 2024Towards Faithful Natural Language Explanations: A Study Using Activation Patching in Large Language Modelscited by 14
  12. 2024Non-Linear Inference Time Intervention: Improving LLM Truthfulnesscited by 9
  13. 2026Causal Attribution via Activation Patchingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.