# Intrinsic Motivation for Encouraging Synergistic Behavior

arXiv: https://arxiv.org/abs/2002.05189 · HTML: https://mlsanity.com/paper/2002.05189/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2020.

## Built on by (most cited first, 13 of 13)

- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy](https://mlsanity.com/paper/2111.05623/) (2021)
- [Stabilize to Act: Learning to Coordinate for Bimanual Manipulation](https://mlsanity.com/paper/2309.01087/) (2023)
- [ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection](https://mlsanity.com/paper/2405.03666/) (2024)
- [AnyBimanual: Transferring Unimanual Policy for General Bimanual Manipulation](https://mlsanity.com/paper/2412.06779/) (2024)
- [ELIGN: Expectation Alignment as a Multi-Agent Intrinsic Reward](https://mlsanity.com/paper/2210.04365/) (2022)
- [MetaVIM: Meta Variationally Intrinsic Motivated Reinforcement Learning for Decentralized Traffic Signal Control](https://mlsanity.com/paper/2101.00746/) (2021)
- [Collaborative Training of Heterogeneous Reinforcement Learning Agents in Environments with Sparse Rewards: What and When to Share?](https://mlsanity.com/paper/2202.12174/) (2022)
- [Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2405.18110/) (2024)
- [DAIR: Disentangled Attention Intrinsic Regularization for Safe and Efficient Bimanual Manipulation](https://mlsanity.com/paper/2106.05907/) (2021)
- [Curiosity-Driven Multi-Agent Exploration with Mixed Objectives](https://mlsanity.com/paper/2210.16468/) (2022)
- [COMBO-Grasp: Learning Constraint-Based Manipulation for Bimanual Occluded Grasping](https://mlsanity.com/paper/2502.08054/) (2025)
- [Multi-State-Action Tokenisation in Decision Transformers for Multi-Discrete Action Spaces](https://mlsanity.com/paper/2407.01310/) (2024)

## Abstract (arXiv)

We study the role of intrinsic motivation as an exploration bias for reinforcement learning in sparse-reward synergistic tasks, which are tasks where multiple agents must work together to achieve a goal they could not individually. Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own. Thus, we propose to incentivize agents to take (joint) actions whose effects cannot be predicted via a composition of the predicted effect for each individual agent. We study two instantiations of this idea, one based on the true states encountered, and another based on a dynamics model trained concurrently with the policy. While the former is simpler, the latter has the benefit of being analytically differentiable with respect to the action taken. We validate our approach in robotic bimanual manipulation and multi-agent locomotion tasks with sparse rewards; we find that our approach yields more efficient learning than both 1) training with only the sparse reward and 2) using the typical surprise-based formulation of intrinsic motivation, which does not bias toward synergistic behavior. Videos are available on the project webpage: https://sites.google.com/view/iclr2020-synergistic.