# Multiagent Soft Q-Learning

arXiv: https://arxiv.org/abs/1804.09817 · HTML: https://mlsanity.com/paper/1804.09817/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2018.

## Built on by (most cited first, 26 of 26)

- [Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.03479/) (2021)
- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning](https://mlsanity.com/paper/2105.15013/) (2021)
- [Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00136/) (2021)
- [Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts](https://mlsanity.com/paper/2105.03363/) (2021)
- [Maximum Entropy Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.10715/) (2023)
- [Multi-Agent Coordination via Multi-Level Communication](https://mlsanity.com/paper/2209.12713/) (2022)
- [Emergence of Theory of Mind Collaboration in Multiagent Systems](https://mlsanity.com/paper/2110.00121/) (2021)
- [Greedy based Value Representation for Optimal Coordination in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2211.12075/) (2022)
- [Learning to Robustly Negotiate Bi-Directional Lane Usage in High-Conflict Driving Scenarios](https://mlsanity.com/paper/2103.12070/) (2021)
- [Multi-agent Actor-Critic with Time Dynamical Opponent Model](https://mlsanity.com/paper/2204.05576/) (2022)
- [Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.15245/) (2022)
- [Learning Explicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning via Polarization Policy Gradient](https://mlsanity.com/paper/2210.05367/) (2022)
- [Finite-Time Error Analysis of Soft Q-Learning: Switching System Approach](https://mlsanity.com/paper/2403.06366/) (2024)
- [Multi-level Advantage Credit Assignment for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2508.06836/) (2025)
- [Recursive Reasoning Graph for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2203.02844/) (2022)
- [Decision-making with Speculative Opponent Models](https://mlsanity.com/paper/2211.11940/) (2022)
- [CURO: Curriculum Learning for Relative Overgeneralization](https://mlsanity.com/paper/2212.02733/) (2022)
- [Negotiated Reasoning: On Provably Addressing Relative Over-Generalization](https://mlsanity.com/paper/2306.05353/) (2023)
- [Deep Reinforcement Learning for the Heat Transfer Control of Pulsating Impinging Jets](https://mlsanity.com/paper/2309.13955/) (2023)
- [Joint Intrinsic Motivation for Coordinated Exploration in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2402.03972/) (2024)
- [PooL: Pheromone-inspired Communication Framework forLarge Scale Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2202.09722/) (2022)
- [Knowing the Past to Predict the Future: Reinforcement Virtual Learning](https://mlsanity.com/paper/2211.01266/) (2022)
- [Mitigating Relative Over-Generalization in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2411.11099/) (2024)
- [Vairiational Stochastic Games](https://mlsanity.com/paper/2503.06037/) (2025)
- [Multi-Agent Decision-Focused Learning via Value-Aware Sequential Communication](https://mlsanity.com/paper/2604.08944/) (2026)

## Abstract (arXiv)

Policy gradient methods are often applied to reinforcement learning in continuous multiagent games. These methods perform local search in the joint-action space, and as we show, they are susceptable to a game-theoretic pathology known as relative overgeneralization. To resolve this issue, we propose Multiagent Soft Q-learning, which can be seen as the analogue of applying Q-learning to continuous controls. We compare our method to MADDPG, a state-of-the-art approach, and show that our method achieves better coordination in multiagent cooperative tasks, converging to better local optima in the joint action space.