# SocialGFs: Learning Social Gradient Fields for Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2405.01839 · HTML: https://mlsanity.com/paper/2405.01839/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 40 references that are in the map.
- First posted on arXiv 3 May 2024.

## Builds on (most cited first, 18 of 18)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1610.03295/) (2016)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Learning Attentional Communication for Multi-Agent Cooperation](https://mlsanity.com/paper/1805.07733/) (2018)
- [Multi-Agent Reinforcement Learning is a Sequence Modeling Problem](https://mlsanity.com/paper/2205.14953/) (2022)
- [Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/1703.06182/) (2017)
- [ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind](https://mlsanity.com/paper/2111.09189/) (2021)
- [GFPose: Learning 3D Human Pose Prior with Gradient Fields](https://mlsanity.com/paper/2212.08641/) (2022)
- [Learning Policy Representations in Multiagent Systems](https://mlsanity.com/paper/1806.06464/) (2018)
- [Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.10423/) (2020)
- [Towards Distraction-Robust Active Visual Tracking](https://mlsanity.com/paper/2106.10110/) (2021)
- [A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings](https://mlsanity.com/paper/2106.09012/) (2021)
- [GraspGF: Learning Score-based Grasping Primitive for Human-assisting Dexterous Grasping](https://mlsanity.com/paper/2309.06038/) (2023)
- [Learning Multi-Agent Coordination for Enhancing Target Coverage in Directional Sensor Networks](https://mlsanity.com/paper/2010.13110/) (2020)
- [Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment](https://mlsanity.com/paper/2106.00517/) (2021)
- [The emergence of division of labor through decentralized social sanctioning](https://mlsanity.com/paper/2208.05568/) (2022)

## Built on by (most cited first, 2 of 2)

- [TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft](https://mlsanity.com/paper/2412.05255/) (2024)
- [Inverse Attention Agents for Multi-Agent Systems](https://mlsanity.com/paper/2410.21794/) (2024)

## Abstract (arXiv)

Multi-agent systems (MAS) need to adaptively cope with dynamic environments, changing agent populations, and diverse tasks. However, most of the multi-agent systems cannot easily handle them, due to the complexity of the state and task space. The social impact theory regards the complex influencing factors as forces acting on an agent, emanating from the environment, other agents, and the agent's intrinsic motivation, referring to the social force. Inspired by this concept, we propose a novel gradient-based state representation for multi-agent reinforcement learning. To non-trivially model the social forces, we further introduce a data-driven method, where we employ denoising score matching to learn the social gradient fields (SocialGFs) from offline samples, e.g., the attractive or repulsive outcomes of each force. During interactions, the agents take actions based on the multi-dimensional gradients to maximize their own rewards. In practice, we integrate SocialGFs into the widely used multi-agent reinforcement learning algorithms, e.g., MAPPO. The empirical results reveal that SocialGFs offer four advantages for multi-agent systems: 1) they can be learned without requiring online interaction, 2) they demonstrate transferability across diverse tasks, 3) they facilitate credit assignment in challenging reward settings, and 4) they are scalable with the increasing number of agents.