How a Cooperative-Override Circuit Suppresses Nash Play in Large Language Models
Not on the map. Why?
Abstract (arXiv)
On the named Prisoner's Dilemma under direct prompting, three larger instruction-tuned models, Llama-3-70B, Qwen2.5-32B, and Qwen2.5-72B, lock at full cooperation, the metric's maximum distance from Nash with zero variance across replicates, while Llama-3-8B plays near-Nash. Opening the models, a logit-lens analysis finds a distributed cooperative override. Intermediate readouts lean toward the Nash action through roughly three quarters of network depth before a late surge toward cooperation, and the final layer settles the contest. The size of that final correction, not the surge, rank-matches chain-of-thought behavior across scale and two architectures. In the 8B the override is a single causally controllable direction in the residual stream; steering it dials the decision, and clamping its component at one position of one layer moves the choice strictly monotonically, Spearman rho = 1.000, with generation fluent. The circuit is lexical. It survives name removal and payoff rescaling but disengages when Cooperate and Defect are replaced with neutral labels, and on 48 payoff-random games with neutral surfaces no model locks cooperative on any dilemma or shows general equilibrium competence. In mixed-model populations a single Nash-playing agent collapses cooperation contagiously. What suppresses Nash play in large language models is a word-triggered circuit rather than missing competence, and it can be measured, bounded, and controlled.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 29 Apr 2026.
Builds on
12 of 17 references.
References · 12
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2023Steering Language Models With Activation Engineeringcited by 824
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
- 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
- 2023Playing repeated games with Large Language Modelscited by 213
- 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
- 2024Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?cited by 62
- 2024Steering Language Models with Game-Theoretic Solverscited by 28
- 2026Persona Vectors in Games: Measuring and Steering Strategies via Activation Vectorscited by 6
- 2025Computational Basis of LLM's Decision Making in Social Simulationcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Collective cooperation without individual fidelity in LLM agents
- 2026Payoff scaling shapes cooperation in LLM agents across languages
- 2025Collaboration and Conflict between Humans and Language Models through the Lens of Game Theory
- 2026Do LLMs Beat Nash? Testing Decentralized Coordination in Self-Play Multi-Agent Games
- 2024Static network structure cannot stabilize cooperation among Large Language Model agents
- 2025Spontaneous Giving and Calculated Greed in Language Models
- 2025AsymPuzl: An Asymmetric Puzzle for multi-agent cooperation
- 2026Sparks of Cooperative Reasoning: LLMs as Strategic Hanabi Agents