mlsanityData to 7 Oct 2026

Hindsight Experience Replay

arXiv 1707.01495 ↗First posted 5 Jul 2017cs.LG cs.AI cs.NEcited by 1,096

Not on the map. Why?

Cited by 159 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task.

Key facts. Cited by 1,096 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 5; 2020: 10; 2021: 184; 2022: 207; 2023: 185; 2024: 152; 2025: 168; 2026: 184.

1,096 papers.

Sort:

By area:

  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  3. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  5. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  6. 2023Learning Interactive Real-World Simulatorscited by 440
  7. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  8. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  9. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  10. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  11. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  12. 2026Reinforcement Learning via Self-Distillationcited by 285
38 more
  1. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  3. 2022Autoformalization with Large Language Modelscited by 252
  4. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  5. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  6. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  7. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  8. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  9. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  10. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  11. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  12. 2022Online Decision Transformercited by 194
  13. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  14. 2021Reinforcement Learning with Prototypical Representationscited by 179
  15. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  16. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  17. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  18. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  19. 2019Goal-conditioned Imitation Learningcited by 159
  20. 2022Human-to-Robot Imitation in the Wildcited by 157
  21. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  22. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  23. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  24. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  25. 2021Discovering and Achieving Goals via World Modelscited by 131
  26. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  27. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  28. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  29. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  30. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  31. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  32. 2021Learning One Representation to Optimize All Rewardscited by 110
  33. 2021APS: Active Pretraining with Successor Featurescited by 109
  34. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  35. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 108
  36. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  37. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  38. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103

50 most cited of 1,096 All 1,096 (JSON)