# Curiosity-Driven Multi-Criteria Hindsight Experience Replay

arXiv: https://arxiv.org/abs/1906.03710 · HTML: https://mlsanity.com/paper/1906.03710/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2019.

## Built on by (most cited first, 11 of 11)

- [Planning Goals for Exploration](https://mlsanity.com/paper/2303.13002/) (2023)
- [Policy Architectures for Compositional Generalization in Control](https://mlsanity.com/paper/2203.05960/) (2022)
- [Learning Dynamic Attribute-factored World Models for Efficient Multi-object Reinforcement Learning](https://mlsanity.com/paper/2307.09205/) (2023)
- [Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks](https://mlsanity.com/paper/2110.02414/) (2021)
- [Learning World Models for Unconstrained Goal Navigation](https://mlsanity.com/paper/2411.02446/) (2024)
- [Automatic Curricula via Expert Demonstrations](https://mlsanity.com/paper/2106.09159/) (2021)
- [Regularity as Intrinsic Reward for Free Play](https://mlsanity.com/paper/2312.01473/) (2023)
- [Towards Teachable Autotelic Agents](https://mlsanity.com/paper/2105.11977/) (2021)
- [Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates](https://mlsanity.com/paper/2202.12967/) (2022)
- [Learning Object-Centered Autotelic Behaviors with Graph Neural Networks](https://mlsanity.com/paper/2204.05141/) (2022)
- [Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations](https://mlsanity.com/paper/2509.15981/) (2025)

## Abstract (arXiv)

Dealing with sparse rewards is a longstanding challenge in reinforcement learning. The recent use of hindsight methods have achieved success on a variety of sparse-reward tasks, but they fail on complex tasks such as stacking multiple blocks with a robot arm in simulation. Curiosity-driven exploration using the prediction error of a learned dynamics model as an intrinsic reward has been shown to be effective for exploring a number of sparse-reward environments. We present a method that combines hindsight with curiosity-driven exploration and curriculum learning in order to solve the challenging sparse-reward block stacking task. We are the first to stack more than two blocks using only sparse reward without human demonstrations.