# GradMAP: Gradient-Based Multi-Agent Proximal Learning for Grid-Edge Flexibility

arXiv: https://arxiv.org/abs/2604.24549 · HTML: https://mlsanity.com/paper/2604.24549/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Model-Augmented Actor-Critic: Backpropagating through Paths](https://mlsanity.com/paper/2005.08068/) (2020)
- [Scalable multi-agent reinforcement learning for distributed control of residential energy flexibility](https://mlsanity.com/paper/2203.03417/) (2022)
- [Gradient Informed Proximal Policy Optimization](https://mlsanity.com/paper/2312.08710/) (2023)
- [Centralised rehearsal of decentralised cooperation: Multi-agent reinforcement learning for the scalable coordination of residential energy flexibility](https://mlsanity.com/paper/2305.18875/) (2023)
- [Comprehensive Modeling of Three-Phase Distribution Systems via the Bus Admittance Matrix](https://mlsanity.com/paper/1705.06782/) (2017)
- [Self-Supervised Learning for Large-Scale Preventive Security Constrained DC Optimal Power Flow](https://mlsanity.com/paper/2311.18072/) (2023)
- [Self-Supervised Learning of Parametric Approximation for Security-Constrained DC-OPF](https://mlsanity.com/paper/2601.13486/) (2026)
- [Non-Iterative Solution for Coordinated Optimal Dispatch via Equivalent Projection-Part I: Theory](https://mlsanity.com/paper/2302.13278/) (2023)
- [Neural Risk Limiting Dispatch in Power Networks: Formulation and Generalization Guarantees](https://mlsanity.com/paper/2402.00772/) (2024)
- [Reparameterization Proximal Policy Optimization](https://mlsanity.com/paper/2508.06214/) (2025)
- [A hard-constrained NN learning framework for rapidly restoring AC-OPF from DC-OPF](https://mlsanity.com/paper/2602.06255/) (2026)

## Built on by (most cited first, 1 of 1)

- [Finite-Sample Probabilistic Safety Certification for AI-Based Grid-Edge Coordination](https://mlsanity.com/paper/2609.28182/) (2026)

## Abstract (arXiv)

Coordinating large populations of grid-edge devices requires learning methods that remain fully decentralised in deployment while still respecting three-phase AC distribution-network physics. This paper proposes gradient-based multi-agent proximal learning (GradMAP) to address this challenge. GradMAP trains independent neural-network policies for each agent without any parameter sharing, and each agent uses only its own local observation for online decision-making without communication. During offline training, GradMAP embeds a differentiable three-phase AC power-flow model in a primal-dual learning loop and uses implicit differentiation to propagate exact network-constraint violations to update the policy parameters. To speed up training, GradMAP reuses expensive environment gradients through a proximal surrogate within a trust region defined in the more direct policy-output (action) space, instead of the probability distribution space used in other works, such as PPO. In case studies with 1,000 agents managing batteries, heat pumps, and controllable generators on the IEEE 123-bus feeder, GradMAP learns decentralised policies that minimise three-phase AC load-flow constraint violations within 15 minutes of training on a single workstation-class NVIDIA RTX PRO 5000 Blackwell 48GB GPU. This is a 3--5x training speed-up over gradient-based self-supervised learning benchmarks and substantially better training efficiency than multi-agent reinforcement-learning benchmarks. In out-of-sample tests, GradMAP also delivers among the lowest operating cost and constraint violations.