mlsanityData to 7 Oct 2026

Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks

arXiv 2110.02414 ↗First posted 5 Oct 2021cs.LG cs.ROcited by 8

Not on the map. Why?

Abstract (arXiv)

Model-based reinforcement learning is a promising learning strategy for practical robotic applications due to its improved data-efficiency versus model-free counterparts. However, current state-of-the-art model-based methods rely on shaped reward signals, which can be difficult to design and implement. To remedy this, we propose a simple model-based method tailored for sparse-reward multi-goal tasks that foregoes the need for complicated reward engineering. This approach, termed Imaginary Hindsight Experience Replay, minimises real-world interactions by incorporating imaginary data into policy updates. To improve exploration in the sparse-reward setting, the policy is trained with standard Hindsight Experience Replay and endowed with curiosity-based intrinsic rewards. Upon evaluation, this approach provides an order of magnitude increase in data-efficiency on average versus the state-of-the-art model-free method in the benchmark OpenAI Gym Fetch Robotics tasks.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 40 of its 46 references that are in the map. First posted on arXiv 5 Oct 2021.

Builds on

40 of 46 references.

References · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2016OpenAI Gymcited by 1,684
  4. 2018World Modelscited by 1,444
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  10. 2020Mastering Atari with Discrete World Modelscited by 990
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
28 more
  1. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  2. 2018Learning Dexterous In-Hand Manipulationcited by 643
  3. 2019Model-Based Reinforcement Learning for Ataricited by 495
  4. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  5. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  6. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  7. 2019Self-Supervised Exploration via Disagreementcited by 263
  8. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  9. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  10. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  11. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  12. 2020Revisiting Fundamentals of Experience Replaycited by 171
  13. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  14. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  15. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  16. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  17. 2018Model-Based Active Explorationcited by 105
  18. 2017A Deeper Look at Experience Replaycited by 104
  19. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  20. 2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 61
  21. 2020PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goalscited by 21
  22. 2019Explicit Explore-Exploit Algorithms in Continuous State Spacescited by 19
  23. 2021MHER: Model-based Hindsight Experience Replaycited by 14
  24. 2019Curiosity-Driven Multi-Criteria Hindsight Experience Replaycited by 11
  25. 2020Episodic Self-Imitation Learning with Hindsightcited by 10
  26. 2018Multi-objective Model-based Policy Search for Data-efficient Learning with Sparse Rewardscited by 6
  27. 2019Addressing Sample Complexity in Visual Tasks Using HER and Hallucinatory GANscited by 6
  28. 2020Soft Hindsight Experience Replaycited by 2

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 3; 2024: 2.

8 papers.

  1. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  2. 2022Relay Hindsight Experience Replay: Self-Guided Continual Reinforcement Learning for Sequential Object Manipulation Tasks with Sparse Rewardscited by 10
  3. 2023Improving Unsupervised Visual Program Inference with Code Rewriting Familiescited by 10
  4. 2024Learning World Models for Unconstrained Goal Navigationcited by 6
  5. 2022Dexterous Robotic Manipulation using Deep Reinforcement Learning and Knowledge Transfer for Complex Sparse Reward-based Taskscited by 5
  6. 2023Magnetic Field-Based Reward Shaping for Goal-Conditioned Reinforcement Learningcited by 2
  7. 2024Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learningcited by 1
  8. 2023MRHER: Model-based Relay Hindsight Experience Replay for Sequential Object Manipulation Tasks with Sparse Rewardscited by 0