# Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing: Reproducible Triggers, Trace Diagnostics, and a Partial Fix

arXiv: https://arxiv.org/abs/2606.09884 · HTML: https://mlsanity.com/paper/2606.09884/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 13 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 7 of 7)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs](https://mlsanity.com/paper/2006.16210/) (2020)
- [Algorithmic Pricing and Algorithmic Collusion](https://mlsanity.com/paper/2504.16592/) (2025)
- [Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2509.09135/) (2025)
- [Tacit algorithmic collusion in deep reinforcement learning guided price competition: A study using EV charge pricing game](https://mlsanity.com/paper/2401.15108/) (2024)
- [By Fair Means or Foul: Quantifying Collusion in a Market Simulation with Deep Reinforcement Learning](https://mlsanity.com/paper/2406.02650/) (2024)
- [Impact of Price Inflation on Algorithmic Collusion Through Reinforcement Learning Agents](https://mlsanity.com/paper/2504.05335/) (2025)

## Abstract (arXiv)

We study two reproducible failure modes of deep multi-agent reinforcement learning in continuous-time pricing markets: (i) tacit cartel formation between competing DDPG agents, and (ii) actor--critic instability at high event rates. We instantiate both inside a single CT-MARL benchmark (Poisson-clocked price updates, observation latency δ, interior-optimum logit demand), show that synchronous DDPG agents reliably trigger Failure Mode 1 with collusion index Delta = 0.69 ± 0.11, and quantify a partial microstructure fix: asynchrony alone cuts collusion by 48% and adding latency drives it to a minimum of Delta = 0.28. The fix has clearly documented costs: it is partial (Delta remains supra-Bertrand), it is non-monotone in δ, and it does not survive Failure Mode 2, which emerges as DDPG critic divergence at λ = 5 and corrupts the phase-diagram cell at (λ=5, δ=1). We accompany the scalar collusion index with trajectory-level trace diagnostics that expose the within-episode signalling collapse and the post-shock non-recovery.