# Complex Robotic Manipulation via Graph-Based Hindsight Goal Generation

arXiv: https://arxiv.org/abs/2007.13486 · HTML: https://mlsanity.com/paper/2007.13486/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2020.

## Built on by (most cited first, 7 of 7)

- [Provably Safe Deep Reinforcement Learning for Robotic Manipulation in Human Environments](https://mlsanity.com/paper/2205.06311/) (2022)
- [Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning](https://mlsanity.com/paper/2305.00286/) (2023)
- [Learning from Symmetry: Meta-Reinforcement Learning with Symmetrical Behaviors and Language Instructions](https://mlsanity.com/paper/2209.10656/) (2022)
- [Contact Energy Based Hindsight Experience Prioritization](https://mlsanity.com/paper/2312.02677/) (2023)
- [Real-Time Adaptive Safety-Critical Control with Gaussian Processes in High-Order Uncertain Models](https://mlsanity.com/paper/2402.18946/) (2024)
- [Learning and reusing primitive behaviours to improve Hindsight Experience Replay sample efficiency](https://mlsanity.com/paper/2310.01827/) (2023)
- [Trajectory Planning of Robotic Manipulator in Dynamic Environment Exploiting DRL](https://mlsanity.com/paper/2403.16652/) (2024)

## Abstract (arXiv)

Reinforcement learning algorithms such as hindsight experience replay (HER) and hindsight goal generation (HGG) have been able to solve challenging robotic manipulation tasks in multi-goal settings with sparse rewards. HER achieves its training success through hindsight replays of past experience with heuristic goals, but under-performs in challenging tasks in which goals are difficult to explore. HGG enhances HER by selecting intermediate goals that are easy to achieve in the short term and promising to lead to target goals in the long term. This guided exploration makes HGG applicable to tasks in which target goals are far away from the object's initial position. However, HGG is not applicable to manipulation tasks with obstacles because the euclidean metric used for HGG is not an accurate distance metric in such environments. In this paper, we propose graph-based hindsight goal generation (G-HGG), an extension of HGG selecting hindsight goals based on shortest distances in an obstacle-avoiding graph, which is a discrete representation of the environment. We evaluated G-HGG on four challenging manipulation tasks with obstacles, where significant enhancements in both sample efficiency and overall success rate are shown over HGG and HER. Videos can be viewed at https://sites.google.com/view/demos-g-hgg/.