# PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1911.00025 · HTML: https://mlsanity.com/paper/1911.00025/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2019.

## Built on by (most cited first, 30 of 47)

- [Residual Pathway Priors for Soft Equivariance Constraints](https://mlsanity.com/paper/2112.01388/) (2021)
- [Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificates](https://mlsanity.com/paper/2101.05436/) (2021)
- [Cooperative Exploration for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2107.11444/) (2021)
- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation](https://mlsanity.com/paper/2108.11550/) (2021)
- [Multi-Agent MDP Homomorphic Networks](https://mlsanity.com/paper/2110.04495/) (2021)
- [GridToPix: Training Embodied Agents with Minimal Supervision](https://mlsanity.com/paper/2105.00931/) (2021)
- [Maximum Class Separation as Inductive Bias in One Matrix](https://mlsanity.com/paper/2206.08704/) (2022)
- [Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents](https://mlsanity.com/paper/2110.05769/) (2021)
- [Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.03319/) (2021)
- [Equivariance Discovery by Learned Parameter-Sharing](https://mlsanity.com/paper/2204.03640/) (2022)
- [Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks](https://mlsanity.com/paper/2210.08001/) (2022)
- [Group-Aware Coordination Graph for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2404.10976/) (2024)
- [Equivariant Networks for Zero-Shot Coordination](https://mlsanity.com/paper/2210.12124/) (2022)
- [Making Vision Transformers Truly Shift-Equivariant](https://mlsanity.com/paper/2305.16316/) (2023)
- [Truly Scale-Equivariant Deep Nets with Fourier Layers](https://mlsanity.com/paper/2311.02922/) (2023)
- [Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach](https://mlsanity.com/paper/2105.08268/) (2021)
- [Relational Reasoning via Set Transformers: Provable Efficiency and Applications to MARL](https://mlsanity.com/paper/2209.09845/) (2022)
- [ELIGN: Expectation Alignment as a Multi-Agent Intrinsic Reward](https://mlsanity.com/paper/2210.04365/) (2022)
- [Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2201.04612/) (2022)
- [Learning Control Admissibility Models with Graph Neural Networks for Multi-Agent Navigation](https://mlsanity.com/paper/2210.09378/) (2022)
- [Inferring Latent Temporal Sparse Coordination Graph for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2403.19253/) (2024)
- [Agent-Centric Representations for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2104.09402/) (2021)
- [Breaking the Curse of Dimensionality in Multiagent State Space: A Unified Agent Permutation Framework](https://mlsanity.com/paper/2203.05285/) (2022)
- [Multi-Flow Transmission in Wireless Interference Networks: A Convergent Graph Learning Approach](https://mlsanity.com/paper/2303.15544/) (2023)
- [rm E(3)-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.11842/) (2023)
- [Communication-Efficient Actor-Critic Methods for Homogeneous Markov Games](https://mlsanity.com/paper/2202.09422/) (2022)
- [Domain Adversarial Graph Convolutional Network Based on RSSI and Crowdsensing for Indoor Localization](https://mlsanity.com/paper/2204.05184/) (2022)
- [Efficient Reinforcement Learning in Resource Allocation Problems Through Permutation Invariant Multi-task Learning](https://mlsanity.com/paper/2102.09361/) (2021)
- [Group Downsampling with Equivariant Anti-aliasing](https://mlsanity.com/paper/2504.17258/) (2025)

## Abstract (arXiv)

Sample efficiency and scalability to a large number of agents are two important goals for multi-agent reinforcement learning systems. Recent works got us closer to those goals, addressing non-stationarity of the environment from a single agent's perspective by utilizing a deep net critic which depends on all observations and actions. The critic input concatenates agent observations and actions in a user-specified order. However, since deep nets aren't permutation invariant, a permuted input changes the critic output despite the environment remaining identical. To avoid this inefficiency, we propose a 'permutation invariant critic' (PIC), which yields identical output irrespective of the agent permutation. This consistent representation enables our model to scale to 30 times more agents and to achieve improvements of test episode reward between 15% to 50% on the challenging multi-agent particle environment (MPE).