mlsanityData to 7 Oct 2026

How a Cooperative-Override Circuit Suppresses Nash Play in Large Language Models

arXiv 2604.27167 ↗First posted 29 Apr 2026cs.GT cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

On the named Prisoner's Dilemma under direct prompting, three larger instruction-tuned models, Llama-3-70B, Qwen2.5-32B, and Qwen2.5-72B, lock at full cooperation, the metric's maximum distance from Nash with zero variance across replicates, while Llama-3-8B plays near-Nash. Opening the models, a logit-lens analysis finds a distributed cooperative override. Intermediate readouts lean toward the Nash action through roughly three quarters of network depth before a late surge toward cooperation, and the final layer settles the contest. The size of that final correction, not the surge, rank-matches chain-of-thought behavior across scale and two architectures. In the 8B the override is a single causally controllable direction in the residual stream; steering it dials the decision, and clamping its component at one position of one layer moves the choice strictly monotonically, Spearman rho = 1.000, with generation fluent. The circuit is lexical. It survives name removal and payoff rescaling but disengages when Cooperate and Defect are replaced with neutral labels, and on 48 payoff-random games with neutral surfaces no model locks cooperative on any dilemma or shows general equilibrium competence. In mixed-model populations a single Nash-playing agent collapses cooperation contagiously. What suppresses Nash play in large language models is a word-triggered circuit rather than missing competence, and it can be measured, bounded, and controlled.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

12 of 17 references.

References · 12

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  4. 2023Steering Language Models With Activation Engineeringcited by 824
  5. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  6. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  7. 2023Playing repeated games with Large Language Modelscited by 213
  8. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
  9. 2024Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?cited by 62
  10. 2024Steering Language Models with Game-Theoretic Solverscited by 28
  11. 2026Persona Vectors in Games: Measuring and Steering Strategies via Activation Vectorscited by 6
  12. 2025Computational Basis of LLM's Decision Making in Social Simulationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Collective cooperation without individual fidelity in LLM agents
  2. 2026Payoff scaling shapes cooperation in LLM agents across languages
  3. 2025Collaboration and Conflict between Humans and Language Models through the Lens of Game Theory
  4. 2026Do LLMs Beat Nash? Testing Decentralized Coordination in Self-Play Multi-Agent Games
  5. 2024Static network structure cannot stabilize cooperation among Large Language Model agents
  6. 2025Spontaneous Giving and Calculated Greed in Language Models
  7. 2025AsymPuzl: An Asymmetric Puzzle for multi-agent cooperation
  8. 2026Sparks of Cooperative Reasoning: LLMs as Strategic Hanabi Agents