# Learning by Playing - Solving Sparse Reward Tasks from Scratch

arXiv: https://arxiv.org/abs/1802.10567 · HTML: https://mlsanity.com/paper/1802.10567/

## Key facts

- Cited by 167 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2018.

## Built on by (most cited first, 30 of 167)

- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [ReFT: Reasoning with Reinforced Fine-Tuning](https://mlsanity.com/paper/2401.08967/) (2024)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning](https://mlsanity.com/paper/2002.08396/) (2020)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [Reinforcement Learning with Action Chunking](https://mlsanity.com/paper/2507.07969/) (2025)
- [The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixels](https://mlsanity.com/paper/2101.02722/) (2021)
- [Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning](https://mlsanity.com/paper/2201.13425/) (2022)
- [Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention](https://mlsanity.com/paper/2104.11203/) (2021)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Latent Plans for Task-Agnostic Offline Reinforcement Learning](https://mlsanity.com/paper/2209.08959/) (2022)
- [Video Prediction Models as Rewards for Reinforcement Learning](https://mlsanity.com/paper/2305.14343/) (2023)
- [For SALE: State-Action Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2306.02451/) (2023)
- [Auxiliary Tasks and Exploration Enable ObjectNav](https://mlsanity.com/paper/2104.04112/) (2021)
- [Asymmetric self-play for automatic goal discovery in robotic manipulation](https://mlsanity.com/paper/2101.04882/) (2021)
- [Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes](https://mlsanity.com/paper/2110.06192/) (2021)
- [CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning](https://mlsanity.com/paper/2212.05711/) (2022)
- [Hierarchical Reinforcement Learning By Discovering Intrinsic Options](https://mlsanity.com/paper/2101.06521/) (2021)
- [On The Effect of Auxiliary Tasks on Representation Dynamics](https://mlsanity.com/paper/2102.13089/) (2021)
- [Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization](https://mlsanity.com/paper/2111.07999/) (2021)
- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [Self-Supervised Online Reward Shaping in Sparse-Reward Environments](https://mlsanity.com/paper/2103.04529/) (2021)
- [Contrastive Difference Predictive Coding](https://mlsanity.com/paper/2310.20141/) (2023)
- [Lifelong Robotic Reinforcement Learning by Retaining Experiences](https://mlsanity.com/paper/2109.09180/) (2021)

## Abstract (arXiv)

We propose Scheduled Auxiliary Control (SAC-X), a new learning paradigm in the context of Reinforcement Learning (RL). SAC-X enables learning of complex behaviors - from scratch - in the presence of multiple sparse reward signals. To this end, the agent is equipped with a set of general auxiliary tasks, that it attempts to learn simultaneously via off-policy RL. The key idea behind our method is that active (learned) scheduling and execution of auxiliary policies allows the agent to efficiently explore its environment - enabling it to excel at sparse reward RL. Our experiments in several challenging robotic manipulation settings demonstrate the power of our approach.