# The Reciprocity Gradient

arXiv: https://arxiv.org/abs/2605.08323 · HTML: https://mlsanity.com/paper/2605.08323/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 44 references that are in the map.
- First posted on arXiv 8 May 2026.

## Builds on (most cited first, 5 of 5)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [Cooperation and Reputation Dynamics with Reinforcement Learning](https://mlsanity.com/paper/2102.07523/) (2021)
- [Bottom-Up Reputation Promotes Cooperation with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2502.01971/) (2025)
- [Talk, Judge, Cooperate: Gossip-Driven Indirect Reciprocity in Self-Interested LLM Agents](https://mlsanity.com/paper/2602.07777/) (2026)

## Abstract (arXiv)

Communication is fundamental to sustaining reciprocity and cooperation in strategic interactions. We identify and formulate the influence attribution problem as the central optimization difficulty inherent in such dynamics for a learning agent: any action or signal the agent emits reshapes the reputations of many third parties along combinatorially branching paths before feeding back into its own future rewards, forcing the agent to account for all of these indirect channels at once when choosing every action. To address this, we introduce the reciprocity gradient, which explicitly backpropagates reward gradients through private estimators of opponents' policies trained from public observations. The gradient flows through the reputation chain itself analytically, rather than being estimated from sampled returns. It jointly optimizes actions and evaluative signals without intrinsic rewards or reward shaping. Empirically, the method recovers near-optimal context-sensitive policies, while sample-based baselines collapse into constant-output policies.