# Efficient Self-Supervised Data Collection for Offline Robot Learning

arXiv: https://arxiv.org/abs/2105.04607 · HTML: https://mlsanity.com/paper/2105.04607/

## Key facts

- Cited by 8 papers in the map (data through 7 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 10 May 2021.

## Builds on (most cited first, 19 of 19)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Count-Based Exploration with Neural Density Models](https://mlsanity.com/paper/1703.01310/) (2017)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Learning to Poke by Poking: Experiential Learning of Intuitive Physics](https://mlsanity.com/paper/1606.07419/) (2016)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning](https://mlsanity.com/paper/1708.02190/) (2017)
- [Learning Robotic Manipulation through Visual Planning and Acting](https://mlsanity.com/paper/1905.04411/) (2019)
- [Directed Exploration for Reinforcement Learning](https://mlsanity.com/paper/1906.07805/) (2019)

## Built on by (most cited first, 8 of 8)

- [Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning](https://mlsanity.com/paper/2201.13425/) (2022)
- [The Challenges of Exploration for Offline Reinforcement Learning](https://mlsanity.com/paper/2201.11861/) (2022)
- [Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping](https://mlsanity.com/paper/2301.02099/) (2023)
- [Distilling Motion Planner Augmented Policies into Visual Control Policies for Robot Manipulation](https://mlsanity.com/paper/2111.06383/) (2021)
- [ARLO: A Framework for Automated Reinforcement Learning](https://mlsanity.com/paper/2205.10416/) (2022)
- [Learning A Simulation-based Visual Policy for Real-world Peg In Unseen Holes](https://mlsanity.com/paper/2205.04297/) (2022)
- [Additive Logistic Mechanism for Privacy-Preserving Self-Supervised Learning](https://mlsanity.com/paper/2205.12430/) (2022)
- [Learning Control by Iterative Inversion](https://mlsanity.com/paper/2211.01724/) (2022)

## Abstract (arXiv)

A practical approach to robot reinforcement learning is to first collect a large batch of real or simulated robot interaction data, using some data collection policy, and then learn from this data to perform various tasks, using offline learning algorithms. Previous work focused on manually designing the data collection policy, and on tasks where suitable policies can easily be designed, such as random picking policies for collecting data about object grasping. For more complex tasks, however, it may be difficult to find a data collection policy that explores the environment effectively, and produces data that is diverse enough for the downstream task. In this work, we propose that data collection policies should actively explore the environment to collect diverse data. In particular, we develop a simple-yet-effective goal-conditioned reinforcement-learning method that actively focuses data collection on novel observations, thereby collecting a diverse data-set. We evaluate our method on simulated robot manipulation tasks with visual inputs and show that the improved diversity of active data collection leads to significant improvements in the downstream learning tasks.