# Beyond Supra-Competitive Outcomes: Collusive Behaviour in Deep Reinforcement Learning for Optimal Execution Games

arXiv: https://arxiv.org/abs/2610.00619 · HTML: https://mlsanity.com/paper/2610.00619/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 8 of 8)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Double Deep Q-Learning for Optimal Execution](https://mlsanity.com/paper/1812.06600/) (2018)
- [A state-constrained differential game arising in optimal portfolio liquidation](https://mlsanity.com/paper/1312.7360/) (2013)
- [Self-Play Q-learners Can Provably Collude in the Iterated Prisoner's Dilemma](https://mlsanity.com/paper/2312.08484/) (2023)
- [Memory-Induced Supra-Competitive Outcomes Between Deep Reinforcement Learning Agents in Optimal Trade Execution](https://mlsanity.com/paper/2605.20348/) (2026)
- [Deviations from the Nash equilibrium in a two-player optimal execution game with reinforcement learning](https://mlsanity.com/paper/2408.11773/) (2024)
- [AI agents in Algorithmic Electricity Markets: On the Emergence of Tacit Collusion](https://mlsanity.com/paper/2608.26896/) (2026)
- [Mitigating Retaliatory Algorithmic Collusion in Repeated Games](https://mlsanity.com/paper/2609.20548/) (2026)

## Abstract (arXiv)

In this paper, we extend earlier findings of supra-competitive outcomes in optimal-execution games by identifying a learned punitive mechanism that deters deviations and provides behavioural evidence of collusion. We investigate this mechanism in a two-player, finite-horizon Almgren-Chriss liquidation game. Independent proximal policy optimisation agents with access to within-episode price and action histories achieve costs below the Nash benchmark. We identify a profitable deviation by training against the mean learned liquidation schedule, then impose its first trade on one of the original agents. The opponent responds by accelerating liquidation. This response more than offsets the deviator's gain in every run and both player roles, while leaving the punisher's average payoff materially unchanged relative to not punishing under the same deviation. The punisher imposes greater losses on the deviator while preserving its own average payoff, despite the availability of more profitable, less punitive liquidation plans. Matching deviations and subsequent additional selling rise and later decline during training, while final policies retain an effective punitive response. We formalise two checks: whether punishment outweighs the gain from deviating, and whether the change in trading behaviour is large enough to account for the loss imposed. Both checks hold for the tested deviation. Together, these findings provide behavioural and economic evidence supporting a collusive interpretation of the learned supra-competitive outcomes.