# A Cooperative Reinforcement Learning Environment for Detecting and Penalizing Betrayal

arXiv: https://arxiv.org/abs/2210.12841 · HTML: https://mlsanity.com/paper/2210.12841/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 23 Oct 2022.

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [Open Problems in Cooperative AI](https://mlsanity.com/paper/2012.08630/) (2020)
- [A Review of Cooperative Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1908.03963/) (2019)
- [Truthful AI: Developing and governing AI that does not lie](https://mlsanity.com/paper/2110.06674/) (2021)
- [Inequity aversion improves cooperation in intertemporal social dilemmas](https://mlsanity.com/paper/1803.08884/) (2018)
- [Towards Explainable Artificial Intelligence](https://mlsanity.com/paper/1909.12072/) (2019)
- [Emergent Communication through Negotiation](https://mlsanity.com/paper/1804.03980/) (2018)
- [The Emergence of Adversarial Communication in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2008.02616/) (2020)
- [Learning Structured Communication for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2002.04235/) (2020)
- [Learning to cooperate: Emergent communication in multi-agent navigation](https://mlsanity.com/paper/2004.01097/) (2020)
- [Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria](https://mlsanity.com/paper/2201.01816/) (2022)
- [Symmetric equilibrium of multi-agent reinforcement learning in repeated prisoner's dilemma](https://mlsanity.com/paper/2101.11861/) (2021)
- [Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective](https://mlsanity.com/paper/2106.05402/) (2021)

## Abstract (arXiv)

In this paper we present a Reinforcement Learning environment that leverages agent cooperation and communication, aimed at detection, learning and ultimately penalizing betrayal patterns that emerge in the behavior of self-interested agents. We provide a description of game rules, along with interesting cases of betrayal and trade-offs that arise. Preliminary experimental investigations illustrate a) betrayal emergence, b) deceptive agents outperforming honest baselines and b) betrayal detection based on classification of behavioral features, which surpasses probabilistic detection baselines. Finally, we propose approaches for penalizing betrayal, list directions for future work and suggest interesting extensions of the environment towards capturing and exploring increasingly complex patterns of social interactions.