# Mitigating Retaliatory Algorithmic Collusion in Repeated Games

arXiv: https://arxiv.org/abs/2609.20548 · HTML: https://mlsanity.com/paper/2609.20548/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 10 of 10)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [Inequity aversion improves cooperation in intertemporal social dilemmas](https://mlsanity.com/paper/1803.08884/) (2018)
- [Algorithmic Collusion by Large Language Models](https://mlsanity.com/paper/2404.00806/) (2024)
- [Strategic Collusion of LLM Agents: Market Division in Multi-Commodity Competitions](https://mlsanity.com/paper/2410.00031/) (2024)
- [Artificial Intelligence and Auction Design](https://mlsanity.com/paper/2202.05947/) (2022)
- [Learning to Mitigate AI Collusion on Economic Platforms](https://mlsanity.com/paper/2202.07106/) (2022)
- [Regulation of Algorithmic Collusion](https://mlsanity.com/paper/2401.15794/) (2024)
- [Regulation of Algorithmic Collusion, Refined: Testing Pessimistic Calibrated Regret](https://mlsanity.com/paper/2501.09740/) (2025)
- [Self-Play Q-learners Can Provably Collude in the Iterated Prisoner's Dilemma](https://mlsanity.com/paper/2312.08484/) (2023)

## Built on by (most cited first, 1 of 1)

- [Beyond Supra-Competitive Outcomes: Collusive Behaviour in Deep Reinforcement Learning for Optimal Execution Games](https://mlsanity.com/paper/2610.00619/) (2026)

## Abstract (arXiv)

Reinforcement learning agents trained to maximize their own reward in repeated interactions can converge to supra-competitive outcomes resembling explicit collusion, without communication or shared design. Existing mitigation approaches are largely tied to specific economic settings, like two-sided platforms and auctions, leaving open how to design interventions for general repeated games. We address this gap by formalizing the connection between empirical observations from prior work on Q-learning collusion and classical theory of Simple Penal Codes (SPCs). We show any non-trivial SPC induces a quantifiable conditional dependence in agents' policies, detectable via the total variation distance between an agent's action distributions across cooperation and defection histories. Building on this connection, we propose CURB (Collusion Unwinding via Reward shaping and Belief injection), a reward-shaping framework that penalizes this Total Variation (TV) distance signal during Q-learning and is guaranteed to convert any SPC fixed point of the dynamics into a trivial one, thus precluding collusive equilibria sustained by punishment threats. Empirically, CURB substantially reduces collusion by Q-learning agents in both Bertrand and Cournot Competition Repeated Games. We further demonstrate that CURB extends to deep Q-network agents in Bertrand competition, suggesting the mechanism generalizes beyond tabular Q-learning.