# Multi-Agent MDP Homomorphic Networks

arXiv: https://arxiv.org/abs/2110.04495 · HTML: https://mlsanity.com/paper/2110.04495/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 9 Oct 2021.

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data](https://mlsanity.com/paper/1807.02547/) (2018)
- [Steerable CNNs](https://mlsanity.com/paper/1612.08498/) (2016)
- [Harmonic Networks: Deep Translation and Rotation Equivariance](https://mlsanity.com/paper/1612.04642/) (2016)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [Graph Convolutional Reinforcement Learning](https://mlsanity.com/paper/1810.09202/) (2018)
- [MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning](https://mlsanity.com/paper/2006.16908/) (2020)
- [Deep Coordination Graphs](https://mlsanity.com/paper/1910.00091/) (2019)
- [A Survey on Traffic Signal Control Methods](https://mlsanity.com/paper/1904.08117/) (2019)
- [Neural Enhanced Belief Propagation on Factor Graphs](https://mlsanity.com/paper/2003.01998/) (2020)
- [Symmetry-Aware Actor-Critic for 3D Molecular Design](https://mlsanity.com/paper/2011.12747/) (2020)
- [PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1911.00025/) (2019)
- [Inference in Probabilistic Graphical Models by Graph Neural Networks](https://mlsanity.com/paper/1803.07710/) (2018)
- [rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch](https://mlsanity.com/paper/1909.01500/) (2019)
- [3D G-CNNs for Pulmonary Nodule Detection](https://mlsanity.com/paper/1804.04656/) (2018)
- [Group Equivariant Generative Adversarial Networks](https://mlsanity.com/paper/2005.01683/) (2020)
- [Augmenting learning using symmetry in a biologically-inspired domain](https://mlsanity.com/paper/1910.00528/) (2019)
- [Goal-conditioned Batch Reinforcement Learning for Rotation Invariant Locomotion](https://mlsanity.com/paper/2004.08356/) (2020)
- [Teaching Deep Convolutional Neural Networks to Play Go](https://mlsanity.com/paper/1412.3409/) (2014)
- [Exploiting Anonymity in Approximate Linear Programming: Scaling to Large Multiagent MDPs (Extended Version)](https://mlsanity.com/paper/1511.09080/) (2015)

## Built on by (most cited first, 25 of 25)

- [Continuous MDP Homomorphisms and Homomorphic Policy Gradient](https://mlsanity.com/paper/2209.07364/) (2022)
- [Maximum Class Separation as Inductive Bias in One Matrix](https://mlsanity.com/paper/2206.08704/) (2022)
- [Equivariant Networks for Zero-Shot Coordination](https://mlsanity.com/paper/2210.12124/) (2022)
- [Continual Vision-based Reinforcement Learning with Group Symmetries](https://mlsanity.com/paper/2210.12301/) (2022)
- [Geometry-aware RL for Manipulation of Varying Shapes and Deformable Objects](https://mlsanity.com/paper/2502.07005/) (2025)
- [Policy Gradient Methods in the Presence of Symmetries and State Abstractions](https://mlsanity.com/paper/2305.05666/) (2023)
- [ESP: Exploiting Symmetry Prior for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2307.16186/) (2023)
- [Understanding when Dynamics-Invariant Data Augmentations Benefit Model-Free Reinforcement Learning Updates](https://mlsanity.com/paper/2310.17786/) (2023)
- [Breaking the Curse of Dimensionality in Multiagent State Space: A Unified Agent Permutation Framework](https://mlsanity.com/paper/2203.05285/) (2022)
- [Equivariant Reinforcement Learning for Quadrotor UAV](https://mlsanity.com/paper/2206.01233/) (2022)
- [rm E(3)-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.11842/) (2023)
- [Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstraction](https://mlsanity.com/paper/2407.13364/) (2024)
- [Equivariant MuZero](https://mlsanity.com/paper/2302.04798/) (2023)
- [Reinforcement Learning with Euclidean Data Augmentation for State-Based Continuous Control](https://mlsanity.com/paper/2410.12983/) (2024)
- [Enriching Disentanglement: From Logical Definitions to Quantitative Metrics](https://mlsanity.com/paper/2305.11512/) (2023)
- [Equivariant Reinforcement Learning Frameworks for Quadrotor Low-Level Control](https://mlsanity.com/paper/2502.20500/) (2025)
- [PEnGUiN: Partially Equivariant Graph NeUral Networks for Sample Efficient MARL](https://mlsanity.com/paper/2503.15615/) (2025)
- [Improving Controller Generalization with Dimensionless Markov Decision Processes](https://mlsanity.com/paper/2504.10006/) (2025)
- [Learning Abstract World Models with a Group-Structured Latent Space](https://mlsanity.com/paper/2506.01529/) (2025)
- [Equivariant Multi-agent Reinforcement Learning for Multimodal Vehicle-to-Infrastructure Systems](https://mlsanity.com/paper/2604.06914/) (2026)
- [Centralized control for multi-agent RL in a complex Real-Time-Strategy game](https://mlsanity.com/paper/2304.13004/) (2023)
- [Enhancing Traffic Signal Control through Model-based Reinforcement Learning and Policy Reuse](https://mlsanity.com/paper/2503.08728/) (2025)
- [Symmetry-Guided Multi-Agent Inverse Reinforcement Learning](https://mlsanity.com/paper/2509.08257/) (2025)
- [Reinforcement Learning Using known Invariances](https://mlsanity.com/paper/2511.03473/) (2025)
- [ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2606.30072/) (2026)

## Abstract (arXiv)

This paper introduces Multi-Agent MDP Homomorphic Networks, a class of networks that allows distributed execution using only local information, yet is able to share experience between global symmetries in the joint state-action space of cooperative multi-agent systems. In cooperative multi-agent systems, complex symmetries arise between different configurations of the agents and their local observations. For example, consider a group of agents navigating: rotating the state globally results in a permutation of the optimal joint policy. Existing work on symmetries in single agent reinforcement learning can only be generalized to the fully centralized setting, because such approaches rely on the global symmetry in the full state-action spaces, and these can result in correspondences across agents. To encode such symmetries while still allowing distributed execution we propose a factorization that decomposes global symmetries into local transformations. Our proposed factorization allows for distributing the computation that enforces global symmetries over local agents and local interactions. We introduce a multi-agent equivariant policy network based on this factorization. We show empirically on symmetric multi-agent problems that globally symmetric distributable policies improve data efficiency compared to non-equivariant baselines.