# Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Games

arXiv: https://arxiv.org/abs/2402.13399 · HTML: https://mlsanity.com/paper/2402.13399/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 101 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 14 of 14)

- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers](https://mlsanity.com/paper/2310.15164/) (2023)
- [Inequity aversion improves cooperation in intertemporal social dilemmas](https://mlsanity.com/paper/1803.08884/) (2018)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [A multi-agent reinforcement learning model of common-pool resource appropriation](https://mlsanity.com/paper/1707.06600/) (2017)
- [If MaxEnt RL is the Answer, What is the Question?](https://mlsanity.com/paper/1910.01913/) (2019)
- [A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings](https://mlsanity.com/paper/2106.09012/) (2021)
- [Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans](https://mlsanity.com/paper/2209.13020/) (2022)
- [Legible Normativity for AI Alignment: The Value of Silly Rules](https://mlsanity.com/paper/1811.01267/) (2018)
- [Model-free conventions in multi-agent reinforcement learning with heterogeneous preferences](https://mlsanity.com/paper/2010.09054/) (2020)
- [A Tale of Three Probabilistic Families: Discriminative, Descriptive and Generative Models](https://mlsanity.com/paper/1810.04261/) (2018)
- [Divide-and-Conquer with Sequential Monte Carlo](https://mlsanity.com/paper/1406.4993/) (2014)
- [That's Mine! Learning Ownership Relations and Norms for Robots](https://mlsanity.com/paper/1812.02576/) (2018)

## Built on by (most cited first, 11 of 11)

- [A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications](https://mlsanity.com/paper/2503.17003/) (2025)
- [Emergence of Social Norms in Generative Agent Societies: Principles and Architecture](https://mlsanity.com/paper/2403.08251/) (2024)
- [The Role of Social Learning and Collective Norm Formation in Fostering Cooperation in LLM Multi-Agent Systems](https://mlsanity.com/paper/2510.14401/) (2025)
- [Resource Rational Contractualism Should Guide AI Alignment](https://mlsanity.com/paper/2506.17434/) (2025)
- [Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value](https://mlsanity.com/paper/2512.03399/) (2025)
- [Extended Inductive Reasoning for Personalized Preference Inference from Behavioral Signals](https://mlsanity.com/paper/2505.18071/) (2025)
- [Operationalising Rawlsian Ethics for Fairness in Norm-Learning Agents](https://mlsanity.com/paper/2412.15163/) (2024)
- [Commitment To Cooperation With Self-Negotiated Contracts](https://mlsanity.com/paper/2607.22750/) (2026)
- [Multi-Agent Norm Perception and Induction in Distributed Healthcare](https://mlsanity.com/paper/2412.18454/) (2024)
- [Realist and Pluralist Conceptions of Intelligence and Their Implications on AI Research](https://mlsanity.com/paper/2511.15282/) (2025)
- [Adaptive Accountability in Networked MAS: Tracing and Mitigating Emergent Norms at Scale](https://mlsanity.com/paper/2512.18561/) (2025)

## Abstract (arXiv)

A universal feature of human societies is the adoption of systems of rules and norms in the service of cooperative ends. How can we build learning agents that do the same, so that they may flexibly cooperate with the human institutions they are embedded in? We hypothesize that agents can achieve this by assuming there exists a shared set of norms that most others comply with while pursuing their individual desires, even if they do not know the exact content of those norms. By assuming shared norms, a newly introduced agent can infer the norms of an existing population from observations of compliance and violation. Furthermore, groups of agents can converge to a shared set of norms, even if they initially diverge in their beliefs about what the norms are. This in turn enables the stability of the normative system: since agents can bootstrap common knowledge of the norms, this leads the norms to be widely adhered to, enabling new entrants to rapidly learn those norms. We formalize this framework in the context of Markov games and demonstrate its operation in a multi-agent environment via approximately Bayesian rule induction of obligative and prohibitive norms. Using our approach, agents are able to rapidly learn and sustain a variety of cooperative institutions, including resource management norms and compensation for pro-social labor, promoting collective welfare while still allowing agents to act in their own interests.