mlsanityData to 14 Oct 2026

Revisiting the Gumbel-Softmax in MADDPG

arXiv 2302.11793 ↗First posted 23 Feb 2023cs.LG cs.AI cs.MAcited by 3

Not on the map. Why?

Abstract (arXiv)

MADDPG is an algorithm in multi-agent reinforcement learning (MARL) that extends the popular single-agent method, DDPG, to multi-agent scenarios. Importantly, DDPG is an algorithm designed for continuous action spaces, where the gradient of the state-action value function exists. For this algorithm to work in discrete action spaces, discrete gradient estimation must be performed. For MADDPG, the Gumbel-Softmax (GS) estimator is used -- a reparameterisation which relaxes a discrete distribution into a similar continuous one. This method, however, is statistically biased, and a recent MARL benchmarking paper suggests that this bias makes MADDPG perform poorly in grid-world situations, where the action space is discrete. Fortunately, many alternatives to the GS exist, boasting a wide range of properties. This paper explores several of these alternatives and integrates them into MADDPG for discrete grid-world scenarios. The corresponding impact on various performance metrics is then measured and analysed. It is found that one of the proposed estimators performs significantly better than the original GS in several tasks, achieving up to 55% higher returns, along with faster convergence.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 33 of its 49 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

33 of 49 references.

References · 33

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  5. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  6. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  8. 2019An Introduction to Variational Autoencoderscited by 1,000
  9. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  10. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  11. 2017Convolutional Networks with Adaptive Inference Graphscited by 148
  12. 2016Discrete Variational Autoencoderscited by 125
21 more
  1. 2020Shared Experience Actor-Critic for Multi-Agent Reinforcement Learningcited by 116
  2. 2017Backpropagation through the Void: Optimizing control variates for black-box gradient estimationcited by 106
  3. 2021Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharingcited by 87
  4. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 85
  5. 2021A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learningcited by 82
  6. 2017REBAR: Low-variance, unbiased gradient estimates for discrete latent variable modelscited by 79
  7. 2017Adversarial Feature Matching for Text Generationcited by 58
  8. 2016GANS for Sequences of Discrete Elements with the Gumbel-softmax Distributioncited by 48
  9. 2015A Game-Theoretic Model and Best-Response Learning Method for Ad Hoc Coordination in Multiagent Systemscited by 44
  10. 2015MuProp: Unbiased Backpropagation for Stochastic Neural Networkscited by 37
  11. 2020Rao-Blackwellizing the Straight-Through Gumbel-Softmax Gradient Estimatorcited by 37
  12. 2020Estimating Gradients for Discrete Random Variables by Sampling without Replacementcited by 31
  13. 2019Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmaxcited by 18
  14. 2017Memory Augmented Neural Networks with Wormhole Connectionscited by 15
  15. 2018Reparameterization Gradient for Non-differentiable Modelscited by 11
  16. 2022Training Discrete Deep Generative Models via Gapped Straight-Through Estimatorcited by 8
  17. 2019Exploiting Causality for Selective Belief Filtering in Dynamic Bayesian Networks (Extended Abstract)cited by 7
  18. 2021CARMS: Categorical-Antithetic-REINFORCE Multi-Sample Gradient Estimatorcited by 4
  19. 2017Hierarchical Multi-scale Attention Networks for Action Recognitioncited by 3
  20. 2018Improved Gradient-Based Optimization Over Discrete Distributionscited by 3
  21. 2019Direct Policy Gradients: Direct Optimization of Policies in Discrete Action Spacescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 26
  2. 2025Altruistic Ride Sharing: A Framework for Fair and Sustainable Urban Mobility via Peer-to-Peer Incentivescited by 0
  3. 2026Comprehensive Comparisons of Reinforcement Learning Algorithms in Autonomous Drivingcited by 0