# Policy Regularization via Noisy Advantage Values for Cooperative Multi-agent Actor-Critic methods

arXiv: https://arxiv.org/abs/2106.14334 · HTML: https://mlsanity.com/paper/2106.14334/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2021.

## Built on by (most cited first, 5 of 5)

- [MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning](https://mlsanity.com/paper/2603.00730/) (2026)
- [Reinforcement learning for traffic signal control in hybrid action space](https://mlsanity.com/paper/2211.12956/) (2022)
- [Optimistic Multi-Agent Policy Gradient](https://mlsanity.com/paper/2311.01953/) (2023)
- [Guidelines for Applying RL and MARL in Cybersecurity Applications](https://mlsanity.com/paper/2503.04262/) (2025)
- [Stimulate the Potential of Robots via Competition](https://mlsanity.com/paper/2403.10487/) (2024)

## Abstract (arXiv)

Recent works have applied the Proximal Policy Optimization (PPO) to the multi-agent cooperative tasks, such as Independent PPO (IPPO); and vanilla Multi-agent PPO (MAPPO) which has a centralized value function. However, previous literature shows that MAPPO may not perform as well as Independent PPO (IPPO) and the Fine-tuned QMIX on Starcraft Multi-Agent Challenge (SMAC). MAPPO-Feature-Pruned (MAPPO-FP) improves the performance of MAPPO by the carefully designed agent-specific features, which may be not friendly to algorithmic utility. By contrast, we find that MAPPO may face the problem of textitThe Policies Overfitting in Multi-agent Cooperation(POMAC), as they learn policies by the sampled advantage values. Then POMAC may lead to updating the multi-agent policies in a suboptimal direction and prevent the agents from exploring better trajectories. In this paper, to mitigate the multi-agent policies overfitting, we propose a novel policy regularization method, which disturbs the advantage values via random Gaussian noise. The experimental results show that our method outperforms the Fine-tuned QMIX, MAPPO-FP, and achieves SOTA on SMAC without agent-specific features. We open-source the code at urlhttps://github.com/hijkzzz/noisy-mappo.