mlsanityData to 7 Oct 2026

MARL with General Utilities via Decentralized Shadow Reward Actor-Critic

arXiv 2106.00543 ↗First posted 29 May 2021stat.ML cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We posit a new mechanism for cooperation in multi-agent reinforcement learning (MARL) based upon any nonlinear function of the team's long-term state-action occupancy measure, i.e., a general utility. This subsumes the cumulative return but also allows one to incorporate risk-sensitivity, exploration, and priors. % We derive the bf Decentralized bf Shadow Reward bf Actor-bf Critic (DSAC) in which agents alternate between policy evaluation (critic), weighted averaging with neighbors (information mixing), and local gradient updates for their policy parameters (actor). DSAC augments the classic critic step by requiring agents to (i) estimate their local occupancy measure in order to (ii) estimate the derivative of the local utility with respect to their occupancy measure, i.e., the "shadow reward". DSAC converges to ε-stationarity in O(1/ε2.5) (Theorem reftheorem:final) or faster O(1/ε2) (Corollary refcorollary:communication) steps with high probability, depending on the amount of communications. We further establish the non-existence of spurious stationary points for this problem, that is, DSAC finds the globally optimal policy (Corollary refcorollary:global). Experiments demonstrate the merits of goals beyond the cumulative return in cooperative MARL.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 28 of its 69 references that are in the map. First posted on arXiv 29 May 2021.

Builds on

28 of 69 references.

References · 28

  1. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  2. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  3. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  4. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  5. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 225
  6. 2017Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 206
  7. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  8. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  9. 2014EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimizationcited by 160
  10. 2017Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learningcited by 114
  11. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  12. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 99
16 more
  1. 2017Coordinated Multi-Agent Imitation Learningcited by 70
  2. 2017Sub-sampled Cubic Regularization for Non-convex Optimizationcited by 63
  3. 2019Biases for Emergent Communication in Multi-agent Reinforcement Learningcited by 48
  4. 2018Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimizationcited by 40
  5. 2020UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learningcited by 32
  6. 2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
  7. 2019Active Exploration in Markov Decision Processescited by 27
  8. 2019Optimization for Reinforcement Learning: From Single Agent to Cooperative Agentscited by 19
  9. 2020Cautious Reinforcement Learning via Distributional Risk in the Dual Domaincited by 18
  10. 2019Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learningcited by 17
  11. 2020Negotiating Team Formation Using Deep Reinforcement Learningcited by 13
  12. 2018Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learningcited by 11
  13. 2020Correcting Experience Replay for Multi-Agent Communicationcited by 8
  14. 2013Decentralized stochastic controlcited by 7
  15. 2019Online Planning for Decentralized Stochastic Control with Partial History Sharingcited by 5
  16. 2016Optimality Conditions for Inventory Controlcited by 3

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 1; 2023: 1.

5 papers.

  1. 2023Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilitiescited by 15
  2. 2021Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guaranteescited by 6
  3. 2021MDPGT: Momentum-based Decentralized Policy Gradient Trackingcited by 6
  4. 2021Provably Efficient Multi-Agent Reinforcement Learning with Fully Decentralized Communicationcited by 5
  5. 2022Offsetting Unequal Competition through RL-assisted Incentive Schemescited by 0