# Learning Altruistic Behaviours in Reinforcement Learning without External Rewards

arXiv: https://arxiv.org/abs/2107.09598 · HTML: https://mlsanity.com/paper/2107.09598/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 72 references that are in the map.
- First posted on arXiv 20 Jul 2021.

## Builds on (most cited first, 30 of 38)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Provably Efficient Maximum Entropy Exploration](https://mlsanity.com/paper/1812.02690/) (2018)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [Efficient Exploration via State Marginal Matching](https://mlsanity.com/paper/1906.05274/) (2019)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [Taming the Noise in Reinforcement Learning via Soft Updates](https://mlsanity.com/paper/1512.08562/) (2015)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2006.07169/) (2020)
- [State Entropy Maximization with Random Encoders for Efficient Exploration](https://mlsanity.com/paper/2102.09430/) (2021)
- [Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1807.09936/) (2018)
- [Multi-Agent Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1907.13220/) (2019)
- [Learning Policy Representations in Multiagent Systems](https://mlsanity.com/paper/1806.06464/) (2018)
- [Shared Autonomy via Hindsight Optimization](https://mlsanity.com/paper/1503.07619/) (2015)
- [Pragmatic-Pedagogic Value Alignment](https://mlsanity.com/paper/1707.06354/) (2017)
- [Evolving intrinsic motivations for altruistic behavior](https://mlsanity.com/paper/1811.05931/) (2018)
- [Too many cooks: Bayesian inference for coordinating multi-agent collaboration](https://mlsanity.com/paper/2003.11778/) (2020)
- [Active Exploration in Markov Decision Processes](https://mlsanity.com/paper/1902.11199/) (2019)

## Built on by (most cited first, 8 of 8)

- [Beyond Joint Demonstrations: Personalized Expert Guidance for Efficient Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2403.08936/) (2024)
- [Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety](https://mlsanity.com/paper/2305.19223/) (2023)
- [Toward Finding Strong Pareto Optimal Policies in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2410.19372/) (2024)
- [Combining Theory of Mind and Kindness for Self-Supervised Human-AI Alignment](https://mlsanity.com/paper/2411.04127/) (2024)
- [When Assisting One Disempowers Another](https://mlsanity.com/paper/2511.04177/) (2025)
- [A Case for AI Safety via Law](https://mlsanity.com/paper/2309.12321/) (2023)
- [We Urgently Need Intrinsically Kind Machines](https://mlsanity.com/paper/2411.04126/) (2024)
- [Understanding Action Effects through Instrumental Empowerment in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2508.15652/) (2025)

## Abstract (arXiv)

Can artificial agents learn to assist others in achieving their goals without knowing what those goals are? Generic reinforcement learning agents could be trained to behave altruistically towards others by rewarding them for altruistic behaviour, i.e., rewarding them for benefiting other agents in a given situation. Such an approach assumes that other agents' goals are known so that the altruistic agent can cooperate in achieving those goals. However, explicit knowledge of other agents' goals is often difficult to acquire. In the case of human agents, their goals and preferences may be difficult to express fully; they might be ambiguous or even contradictory. Thus, it is beneficial to develop agents that do not depend on external supervision and learn altruistic behaviour in a task-agnostic manner. We propose to act altruistically towards other agents by giving them more choice and allowing them to achieve their goals better. Some concrete examples include opening a door for others or safeguarding them to pursue their objectives without interference. We formalize this concept and propose an altruistic agent that learns to increase the choices another agent has by preferring to maximize the number of states that the other agent can reach in its future. We evaluate our approach in three different multi-agent environments where another agent's success depends on altruistic behaviour. Finally, we show that our unsupervised agents can perform comparably to agents explicitly trained to work cooperatively, in some cases even outperforming them.