# Learning Dexterous Manipulation from Suboptimal Experts

arXiv: https://arxiv.org/abs/2010.08587 · HTML: https://mlsanity.com/paper/2010.08587/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2020.

## Built on by (most cited first, 23 of 23)

- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [A System for General In-Hand Object Re-Orientation](https://mlsanity.com/paper/2111.03043/) (2021)
- [Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes](https://mlsanity.com/paper/2211.11744/) (2022)
- [DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation](https://mlsanity.com/paper/2211.09423/) (2022)
- [Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes](https://mlsanity.com/paper/2110.06192/) (2021)
- [Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning](https://mlsanity.com/paper/2111.03062/) (2021)
- [DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics](https://mlsanity.com/paper/2304.03223/) (2023)
- [Robot Learning of Mobile Manipulation with Reachability Behavior Priors](https://mlsanity.com/paper/2203.04051/) (2022)
- [DexTransfer: Real World Multi-fingered Dexterous Grasping with Minimal Human Demonstrations](https://mlsanity.com/paper/2209.14284/) (2022)
- [Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics](https://mlsanity.com/paper/2107.09822/) (2021)
- [Collect & Infer -- a fresh look at data-efficient Reinforcement Learning](https://mlsanity.com/paper/2108.10273/) (2021)
- [Replay across Experiments: A Natural Extension of Off-Policy RL](https://mlsanity.com/paper/2311.15951/) (2023)
- [How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation](https://mlsanity.com/paper/2205.03353/) (2022)
- [DexCatch: Learning to Catch Arbitrary Objects with Dexterous Hands](https://mlsanity.com/paper/2310.08809/) (2023)
- [SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration](https://mlsanity.com/paper/2211.13743/) (2022)
- [Forgetting and Imbalance in Robot Lifelong Learning with Off-policy Data](https://mlsanity.com/paper/2204.05893/) (2022)
- [Learning to Transfer In-Hand Manipulations Using a Greedy Shape Curriculum](https://mlsanity.com/paper/2303.12726/) (2023)
- [Dexterous In-hand Manipulation by Guiding Exploration with Simple Sub-skill Controllers](https://mlsanity.com/paper/2303.03533/) (2023)
- [RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands](https://mlsanity.com/paper/2408.11048/) (2024)
- [Back to Reality for Imitation Learning](https://mlsanity.com/paper/2111.12867/) (2021)
- [Dexterous Manipulation Based on Prior Dexterous Grasp Pose Knowledge](https://mlsanity.com/paper/2412.15587/) (2024)
- [Evaluating the Effectiveness of Corrective Demonstrations and a Low-Cost Sensor for Dexterous Manipulation](https://mlsanity.com/paper/2204.07631/) (2022)
- [Curriculum Is More Influential Than Haptic Information During Reinforcement Learning of Object Manipulation Against Gravity](https://mlsanity.com/paper/2407.09986/) (2024)

## Abstract (arXiv)

Learning dexterous manipulation in high-dimensional state-action spaces is an important open challenge with exploration presenting a major bottleneck. Although in many cases the learning process could be guided by demonstrations or other suboptimal experts, current RL algorithms for continuous action spaces often fail to effectively utilize combinations of highly off-policy expert data and on-policy exploration data. As a solution, we introduce Relative Entropy Q-Learning (REQ), a simple policy iteration algorithm that combines ideas from successful offline and conventional RL algorithms. It represents the optimal policy via importance sampling from a learned prior and is well-suited to take advantage of mixed data distributions. We demonstrate experimentally that REQ outperforms several strong baselines on robotic manipulation tasks for which suboptimal experts are available. We show how suboptimal experts can be constructed effectively by composing simple waypoint tracking controllers, and we also show how learned primitives can be combined with waypoint controllers to obtain reference behaviors to bootstrap a complex manipulation task on a simulated bimanual robot with human-like hands. Finally, we show that REQ is also effective for general off-policy RL, offline RL, and RL from demonstrations. Videos and further materials are available at sites.google.com/view/rlfse.