mlsanityData to 14 Oct 2026

Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation

arXiv 2112.00597 ↗First posted 1 Dec 2021cs.RO stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Complex sequential tasks in continuous-control settings often require agents to successfully traverse a set of "narrow passages" in their state space. Solving such tasks with a sparse reward in a sample-efficient manner poses a challenge to modern reinforcement learning (RL) due to the associated long-horizon nature of the problem and the lack of sufficient positive signal during learning. Various tools have been applied to address this challenge. When available, large sets of demonstrations can guide agent exploration. Hindsight relabelling on the other hand does not require additional sources of information. However, existing strategies explore based on task-agnostic goal distributions, which can render the solution of long-horizon tasks impractical. In this work, we extend hindsight relabelling mechanisms to guide exploration along task-specific distributions implied by a small set of successful demonstrations. We evaluate the approach on four complex, single and dual arm, robotics manipulation tasks against strong suitable baselines. The method requires far fewer demonstrations to solve all tasks and achieves a significantly higher overall performance as task complexity increases. Finally, we investigate the robustness of the proposed solution with respect to the quality of input representations and the number of demonstrations.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 38 of its 55 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

38 of 55 references.

References · 38

  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  2. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  3. 2017Hindsight Experience Replaycited by 1,136
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  6. 2019Solving Rubik's Cube with a Robot Handcited by 734
  7. 2016The Option-Critic Architecturecited by 448
  8. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  9. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  12. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
26 more
  1. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  2. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  3. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  4. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  5. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  6. 2019Goal-conditioned Imitation Learningcited by 162
  7. 2019Temporal Cycle-Consistency Learningcited by 154
  8. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  9. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  10. 2018Playing hard exploration games by watching YouTubecited by 99
  11. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 95
  12. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 80
  13. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 68
  14. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 60
  15. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  16. 2020Deep Imitation Learning for Bimanual Robotic Manipulationcited by 50
  17. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 44
  18. 2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 36
  19. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 33
  20. 2020Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learningcited by 31
  21. 2020Data-efficient Hindsight Off-policy Option Learningcited by 31
  22. 2021Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Studycited by 30
  23. 2020Residual Learning from Demonstration: Adapting DMPs for Contact-rich Manipulationcited by 28
  24. 2019Self-supervised Learning of Image Embedding for Continuous Controlcited by 16
  25. 2018One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RLcited by 11
  26. 2020Supervised Learning and Reinforcement Learning of Feedback Models for Reactive Behaviors: Tactile Feedback Testbedcited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 8; 2024: 2; 2025: 1.

13 papers.

Sort:

By area:

  1. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  2. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 111
  3. 2023RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitationcited by 90
  4. 2023Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedbackcited by 13
  5. 2023Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedbackcited by 13
  6. 2023Replay across Experiments: A Natural Extension of Off-Policy RLcited by 10
  7. 2023Intrinsic Language-Guided Exploration for Complex Long-Horizon Robotic Manipulation Taskscited by 8
  8. 2022E-MAPP: Efficient Multi-Agent Reinforcement Learning with Parallel Program Guidancecited by 6
  9. 2023Learning from Pixels with Expert Observationscited by 3
  10. 2022CIM: Constrained Intrinsic Motivation for Sparse-Reward Continuous Controlcited by 1
  11. 2024Real-World Fluid Directed Rigid Body Control via Deep Reinforcement Learningcited by 1
  12. 2023RObotic MAnipulation Network (ROMAN) -- Hybrid Hierarchical Learning for Solving Complex Sequential Taskscited by 0
1 more
  1. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0