# Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2104.06655 · HTML: https://mlsanity.com/paper/2104.06655/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 12 of 12)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1702.08887/) (2017)
- [Off-Policy Multi-Agent Decomposed Policy Gradients](https://mlsanity.com/paper/2007.12322/) (2020)
- [Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning](https://mlsanity.com/paper/2002.03939/) (2020)
- [Analysing Factorizations of Action-Value Networks for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1902.07497/) (2019)

## Built on by (most cited first, 10 of 10)

- [Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients](https://mlsanity.com/paper/2201.01247/) (2022)
- [Safe reinforcement learning for multi-energy management systems with known constraint functions](https://mlsanity.com/paper/2207.03830/) (2022)
- [Coordination Failure in Cooperative Offline MARL](https://mlsanity.com/paper/2407.01343/) (2024)
- [Joint Resource Management for Energy-efficient UAV-assisted SWIPT-MEC: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2505.03230/) (2025)
- [An adaptive safety layer with hard constraints for safe reinforcement learning in multi-energy management systems](https://mlsanity.com/paper/2304.08897/) (2023)
- [Safety Constrained Multi-Agent Reinforcement Learning for Active Voltage Control](https://mlsanity.com/paper/2405.08443/) (2024)
- [The StarCraft Multi-Agent Challenges+ : Learning of Multi-Stage Tasks and Environmental Factors without Precise Reward Functions](https://mlsanity.com/paper/2207.02007/) (2022)
- [Policies over Poses: Reinforcement Learning based Distributed Pose-Graph Optimization for Multi-Robot SLAM](https://mlsanity.com/paper/2510.22740/) (2025)
- [A Negotiation-Based Multi-Agent Reinforcement Learning Approach for Dynamic Scheduling of Reconfigurable Manufacturing Systems](https://mlsanity.com/paper/2511.07707/) (2025)
- [ARAC: Adaptive Regularized Multi-Agent Soft Actor-Critic in Graph-Structured Adversarial Games](https://mlsanity.com/paper/2511.08412/) (2025)

## Abstract (arXiv)

Deep reinforcement learning methods have shown great performance on many challenging cooperative multi-agent tasks. Two main promising research directions are multi-agent value function decomposition and multi-agent policy gradients. In this paper, we propose a new decomposed multi-agent soft actor-critic (mSAC) method, which effectively combines the advantages of the aforementioned two methods. The main modules include decomposed Q network architecture, discrete probabilistic policy and counterfactual advantage function (optinal). Theoretically, mSAC supports efficient off-policy learning and addresses credit assignment problem partially in both discrete and continuous action spaces. Tested on StarCraft II micromanagement cooperative multiagent benchmark, we empirically investigate the performance of mSAC against its variants and analyze the effects of the different components. Experimental results demonstrate that mSAC significantly outperforms policy-based approach COMA, and achieves competitive results with SOTA value-based approach Qmix on most tasks in terms of asymptotic perfomance metric. In addition, mSAC achieves pretty good results on large action space tasks, such as 2c_vs_64zg and MMM2.