# Payoff scaling shapes cooperation in LLM agents across languages

arXiv: https://arxiv.org/abs/2601.19082 · HTML: https://mlsanity.com/paper/2601.19082/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 55 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 15 of 15)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Playing repeated games with Large Language Models](https://mlsanity.com/paper/2305.16867/) (2023)
- [Multi-Agent Risks from Advanced AI](https://mlsanity.com/paper/2502.14143/) (2025)
- [Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis](https://mlsanity.com/paper/2312.05488/) (2023)
- [Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framing](https://mlsanity.com/paper/2309.05898/) (2023)
- [When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmas](https://mlsanity.com/paper/2505.19212/) (2025)
- [When to (or not to) trust intelligent machines: Insights from an evolutionary game theory analysis of trust in repeated games](https://mlsanity.com/paper/2007.11338/) (2020)
- [Media and responsible AI governance: a game-theoretic and LLM analysis](https://mlsanity.com/paper/2503.09858/) (2025)
- [Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents](https://mlsanity.com/paper/2504.08640/) (2025)
- [Inferring Strategies from Observations in Long Iterated Prisoner's Dilemma Experiments](https://mlsanity.com/paper/2202.04171/) (2022)
- [Mapping Human Anti-collusion Mechanisms to Multi-agent AI Systems](https://mlsanity.com/paper/2601.00360/) (2026)
- [How large language models judge and influence human cooperation](https://mlsanity.com/paper/2507.00088/) (2025)
- [Functional Dynamics by Intention Recognition in Iterated Games](https://mlsanity.com/paper/1810.01740/) (2018)

## Built on by (most cited first, 1 of 1)

- [Why LLM Agents Favor Their Group: Stakes, Observed Norms, and Reputation](https://mlsanity.com/paper/2610.11008/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed as autonomous agents that negotiate, coordinate, and act on behalf of users. Whether they cooperate in such settings is no longer just an academic question, but a central issue for AI governance. We approach it from a strategic-behaviour angle, asking how two everyday levers - the size of what is at stake, and the language in which the interaction is described - shape the strategies LLMs adopt in a repeated Prisoner's Dilemma. Rather than reading cooperation off raw action counts, we train supervised classifiers to recognise the canonical strategies of repeated games (always cooperate, always defect, Tit-for-Tat, Win-Stay-Lose-Shift) and use them as a lens onto LLM behaviour. To know what the strategy distribution should look like under the same payoffs, we derive an evolutionary game theory (EGT) baseline and compare it with the LLM data. The two outcomes disagree in a revealing way: as stakes grow, evolutionary theory predicts that defection should take over the population, yet LLMs move in the opposite direction, becoming more cooperative - a signature, we argue, of alignment training and the human-like reasoning patterns LLMs inherit from their training data. We further show that this picture is not particular to frontier-scale, proprietary models: it also occurs with three open-weight smaller LLMs. Overall, our analysis highlights that payoff design and linguistic framing are powerful but under-explored levers for steering LLM behaviour, with direct implications for evaluating, aligning, and governing multi-agent AI systems deployed in high-stakes, multilingual environments.