# A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings

arXiv: https://arxiv.org/abs/2106.09012 · HTML: https://mlsanity.com/paper/2106.09012/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2021.

## Built on by (most cited first, 24 of 24)

- [Melting Pot 2.0](https://mlsanity.com/paper/2211.13746/) (2022)
- [A Review of Cooperation in Multi-agent Learning](https://mlsanity.com/paper/2312.05162/) (2023)
- [Emergent Bartering Behaviour in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.06760/) (2022)
- [Robustness and Adaptability of Reinforcement Learning based Cooperative Autonomous Driving in Mixed-autonomy Traffic](https://mlsanity.com/paper/2202.00881/) (2022)
- [Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmas](https://mlsanity.com/paper/2402.17270/) (2024)
- [A Pragmatic View of AI Personhood](https://mlsanity.com/paper/2510.26396/) (2025)
- [Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia](https://mlsanity.com/paper/2512.03318/) (2025)
- [Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Games](https://mlsanity.com/paper/2402.13399/) (2024)
- [Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt](https://mlsanity.com/paper/2505.05197/) (2025)
- [The emergence of division of labor through decentralized social sanctioning](https://mlsanity.com/paper/2208.05568/) (2022)
- [Investigating the Impact of Direct Punishment on the Emergence of Cooperation in Multi-Agent Reinforcement Learning Systems](https://mlsanity.com/paper/2301.08278/) (2023)
- [Normative Modules: A Generative Agent Architecture for Learning Norms that Supports Multi-Agent Cooperation](https://mlsanity.com/paper/2405.19328/) (2024)
- [Artificial Intelligence for Collective Intelligence: A National-Scale Research Strategy](https://mlsanity.com/paper/2411.06211/) (2024)
- [Agile Flight Emerges from Multi-Agent Competitive Racing](https://mlsanity.com/paper/2512.11781/) (2025)
- [Social learning spontaneously emerges by searching optimal heuristics with deep reinforcement learning](https://mlsanity.com/paper/2204.12371/) (2022)
- [Developing, Evaluating and Scaling Learning Agents in Multi-Agent Environments](https://mlsanity.com/paper/2209.10958/) (2022)
- [Learning Roles with Emergent Social Value Orientations](https://mlsanity.com/paper/2301.13812/) (2023)
- [SocialGFs: Learning Social Gradient Fields for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2405.01839/) (2024)
- [Incentives to Build Houses, Trade Houses, or Trade House Building Skills in Simulated Worlds under Various Governing Systems or Institutions: Comparing Multi-agent Reinforcement Learning to Generative Agent-based Model](https://mlsanity.com/paper/2411.17724/) (2024)
- [Emergent Dominance Hierarchies in Reinforcement Learning Agents](https://mlsanity.com/paper/2401.12258/) (2024)
- [Enhancing Cooperation through Selective Interaction and Long-term Experiences in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2405.02654/) (2024)
- [Operationalising Rawlsian Ethics for Fairness in Norm-Learning Agents](https://mlsanity.com/paper/2412.15163/) (2024)
- [Large Legislative Models: Towards Efficient AI Policymaking in Economic Simulations](https://mlsanity.com/paper/2410.08345/) (2024)
- [Learning Incentive Structures for Cooperative Resilience in Multi-Agent Systems under Social Dilemmas](https://mlsanity.com/paper/2601.22292/) (2026)

## Abstract (arXiv)

Society is characterized by the presence of a variety of social norms: collective patterns of sanctioning that can prevent miscoordination and free-riding. Inspired by this, we aim to construct learning dynamics where potentially beneficial social norms can emerge. Since social norms are underpinned by sanctioning, we introduce a training regime where agents can access all sanctioning events but learning is otherwise decentralized. This setting is technologically interesting because sanctioning events may be the only available public signal in decentralized multi-agent systems where reward or policy-sharing is infeasible or undesirable. To achieve collective action in this setting we construct an agent architecture containing a classifier module that categorizes observed behaviors as approved or disapproved, and a motivation to punish in accord with the group. We show that social norms emerge in multi-agent systems containing this agent and investigate the conditions under which this helps them achieve socially beneficial outcomes.