mlsanityData to 7 Oct 2026

Scalable Constrained Multi-Agent Reinforcement Learning via State Augmentation and Consensus for Separable Dynamics

arXiv 2605.30461 ↗First posted 28 May 2026cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

We present a distributed approach for constrained Multi-Agent Reinforcement Learning (MARL) that combines state-augmented policy learning with distributed consensus over dual variables. Our method targets systems where agents have separable dynamics but must coordinate to satisfy global resource constraints, a setting in which, as we demonstrate empirically, independent learning fails to produce feasible solutions because agents cannot determine appropriate individual contributions toward collective constraint satisfaction. The key technical contribution is showing that lightweight neighbor-to-neighbor consensus over Lagrange multipliers suffices for globally coordinated constraint enforcement while preserving the scalability of independent training. Each agent learns a single augmented policy offline, conditioned on both its local state and a dual variable encoding constraint feedback. During execution, agents reach agreement on this dual variable through local communication alone. We prove that under mild connectivity assumptions, the consensus error among agents' multipliers is bounded, and show that this translates to a bounded constraint violation that decreases with graph connectivity and the number of consensus rounds. Unlike centralized training with decentralized execution (CTDE) approaches, whose complexity grows at least quadratically with agent count, our method scales linearly in both training and execution. Experiments on smart grid demand response demonstrate that consensus coordination is essential for feasibility: without it, agents satisfy grid capacity constraints only by indefinitely postponing demand, a degenerate non-solution. With consensus, agents converge to a shared dual variable and satisfy both grid constraints and demand fulfillment, scaling to thousands of agents while CTDE baselines are limited to dozens.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 61 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

32 of 61 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2013Intriguing properties of neural networkscited by 5,114
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  7. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  8. 2020Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?cited by 281
  9. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  10. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  11. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
  12. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
20 more
  1. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 75
  2. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 61
  3. 2020Multi-agent Reinforcement Learning for Networked System Controlcited by 51
  4. 2020Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Rewardcited by 46
  5. 2021State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewardscited by 37
  6. 2021Multi-Agent Constrained Policy Optimisationcited by 32
  7. 2020CityLearn: Standardizing Research in Multi-Agent Reinforcement Learning for Demand Response and Urban Energy Managementcited by 27
  8. 2023Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilitiescited by 15
  9. 2025Multi-Agent Reinforcement Learning for Resources Allocation Optimization: A Surveycited by 14
  10. 2020PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Controlcited by 13
  11. 2023Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learningcited by 13
  12. 2022Stability Constrained Reinforcement Learning for Decentralized Real-Time Voltage Controlcited by 8
  13. 2023Structured Neural-PI Control with End-to-End Stability and Output Tracking Guaranteescited by 5
  14. 2025A Survey of Multi Agent Reinforcement Learning: Federated Learning and Cooperative and Noncooperative Decentralized Regimescited by 5
  15. 2023Bridging Transient and Steady-State Performance in Voltage Control: A Reinforcement Learning Approach with Safe Gradient Flowcited by 3
  16. 2024Multi-agent assignment via state augmented reinforcement learningcited by 3
  17. 2025Consensus-based Decentralized Multi-agent Reinforcement Learning for Random Access Network Optimizationcited by 3
  18. 2023Multi-Agent Reinforcement Learning for Fast-Timescale Demand Response of Residential Loadscited by 2
  19. 2024Decentralized Optimization with Coupled Constraintscited by 2
  20. 2024Globally-Constrained Decentralized Optimization with Variable Couplingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Learning Infinite-Horizon Average-Reward CMDPs via State Augmentationcited by 0