# Regularity as Intrinsic Reward for Free Play

arXiv: https://arxiv.org/abs/2312.01473 · HTML: https://mlsanity.com/paper/2312.01473/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 51 references that are in the map.
- First posted on arXiv 3 Dec 2023.

## Builds on (most cited first, 30 of 30)

- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [Planning to Explore via Self-Supervised World Models](https://mlsanity.com/paper/2005.05960/) (2020)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [Self-Supervised Exploration via Disagreement](https://mlsanity.com/paper/1906.04161/) (2019)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [Discovering and Achieving Goals via World Models](https://mlsanity.com/paper/2110.09514/) (2021)
- [Sample-efficient Cross-Entropy Method for Real-time Planning](https://mlsanity.com/paper/2008.06389/) (2020)
- [CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/1810.06284/) (2018)
- [Asymmetric self-play for automatic goal discovery in robotic manipulation](https://mlsanity.com/paper/2101.04882/) (2021)
- [Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning](https://mlsanity.com/paper/1912.11032/) (2019)
- [Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning](https://mlsanity.com/paper/2107.12544/) (2021)
- [Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulation](https://mlsanity.com/paper/2206.11403/) (2022)
- [Active World Model Learning with Progress Curiosity](https://mlsanity.com/paper/2007.07853/) (2020)
- [Planning Goals for Exploration](https://mlsanity.com/paper/2303.13002/) (2023)
- [Capturing the objects of vision with neural networks](https://mlsanity.com/paper/2109.03351/) (2021)
- [Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration](https://mlsanity.com/paper/2109.08603/) (2021)
- [Sparse Reward Exploration via Novelty Search and Emitters](https://mlsanity.com/paper/2102.03140/) (2021)
- [Curiosity-Driven Multi-Criteria Hindsight Experience Replay](https://mlsanity.com/paper/1906.03710/) (2019)
- [Factored World Models for Zero-Shot Generalization in Robotic Manipulation](https://mlsanity.com/paper/2202.05333/) (2022)

## Built on by (most cited first, 5 of 5)

- [Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability](https://mlsanity.com/paper/2601.18778/) (2026)
- [SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models](https://mlsanity.com/paper/2503.01584/) (2025)
- [ReaLJam: Real-Time Human-AI Music Jamming with Reinforcement Learning-Tuned Transformers](https://mlsanity.com/paper/2502.21267/) (2025)
- [Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learning](https://mlsanity.com/paper/2507.14748/) (2025)
- [VendiRL: A Framework for Self-Supervised Reinforcement Learning of Diversely Diverse Skills](https://mlsanity.com/paper/2509.02930/) (2025)

## Abstract (arXiv)

We propose regularity as a novel reward signal for intrinsically-motivated reinforcement learning. Taking inspiration from child development, we postulate that striving for structure and order helps guide exploration towards a subspace of tasks that are not favored by naive uncertainty-based intrinsic rewards. Our generalized formulation of Regularity as Intrinsic Reward (RaIR) allows us to operationalize it within model-based reinforcement learning. In a synthetic environment, we showcase the plethora of structured patterns that can emerge from pursuing this regularity objective. We also demonstrate the strength of our method in a multi-object robotic manipulation environment. We incorporate RaIR into free play and use it to complement the model's epistemic uncertainty as an intrinsic reward. Doing so, we witness the autonomous construction of towers and other regular structures during free play, which leads to a substantial improvement in zero-shot downstream task performance on assembly tasks.