# Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance

arXiv: https://arxiv.org/abs/2212.09902 · HTML: https://mlsanity.com/paper/2212.09902/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 59 references that are in the map.
- First posted on arXiv 19 Dec 2022.

## Builds on (most cited first, 30 of 30)

- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [End-to-End Robotic Reinforcement Learning without Reward Engineering](https://mlsanity.com/paper/1904.07854/) (2019)
- [The Ingredients of Real-World Robotic Reinforcement Learning](https://mlsanity.com/paper/2004.12570/) (2020)
- [Learning to be Safe: Deep RL with a Safety Critic](https://mlsanity.com/paper/2010.14603/) (2020)
- [Dropout Q-Functions for Doubly Efficient Reinforcement Learning](https://mlsanity.com/paper/2110.02034/) (2021)
- [Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention](https://mlsanity.com/paper/2104.11203/) (2021)
- [Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning](https://mlsanity.com/paper/1711.06782/) (2017)
- [Learning Dexterous Grasping with Object-Centric Visual Affordances](https://mlsanity.com/paper/2009.01439/) (2020)
- [Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost](https://mlsanity.com/paper/1810.06045/) (2018)
- [Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition](https://mlsanity.com/paper/1805.11686/) (2018)
- [Learning Multimodal Rewards from Rankings](https://mlsanity.com/paper/2109.12750/) (2021)
- [Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger](https://mlsanity.com/paper/2108.09779/) (2021)
- [Guiding Policies with Language via Meta-Learning](https://mlsanity.com/paper/1811.07882/) (2018)
- [Learning Dexterous Manipulation Policies from Experience and Imitation](https://mlsanity.com/paper/1611.05095/) (2016)
- [MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning](https://mlsanity.com/paper/2107.07184/) (2021)
- [Learning Multi-Level Hierarchies with Hindsight](https://mlsanity.com/paper/1712.00948/) (2017)
- [Learning Precise 3D Manipulation from Multiple Uncalibrated Cameras](https://mlsanity.com/paper/2002.09107/) (2020)
- [Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system](https://mlsanity.com/paper/1803.10371/) (2018)
- [Including Uncertainty when Learning from Human Corrections](https://mlsanity.com/paper/1806.02454/) (2018)

## Built on by (most cited first, 23 of 23)

- [DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation](https://mlsanity.com/paper/2403.07788/) (2024)
- [DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping](https://mlsanity.com/paper/2502.20900/) (2025)
- [A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning](https://mlsanity.com/paper/2509.15937/) (2025)
- [Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning](https://mlsanity.com/paper/2310.15145/) (2023)
- [DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation](https://mlsanity.com/paper/2505.24853/) (2025)
- [Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning](https://mlsanity.com/paper/2303.01488/) (2023)
- [Universal Visual Decomposer: Long-Horizon Manipulation Made Easy](https://mlsanity.com/paper/2310.08581/) (2023)
- [Continuously Improving Mobile Manipulation with Autonomous Real-World RL](https://mlsanity.com/paper/2409.20568/) (2024)
- [Dexterous Manipulation through Imitation Learning: A Survey](https://mlsanity.com/paper/2504.03515/) (2025)
- [REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation](https://mlsanity.com/paper/2309.03322/) (2023)
- [Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback](https://mlsanity.com/paper/2310.20608/) (2023)
- [From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning](https://mlsanity.com/paper/2603.10263/) (2026)
- [Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoids](https://mlsanity.com/paper/2508.12252/) (2025)
- [Test-Time Regret Minimization in Meta Reinforcement Learning](https://mlsanity.com/paper/2406.02282/) (2024)
- [Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning](https://mlsanity.com/paper/2505.12278/) (2025)
- [SimLauncher: Launching Sample-Efficient Real-world Robotic Reinforcement Learning via Simulation Pre-training](https://mlsanity.com/paper/2507.04452/) (2025)
- [When Learning Is Out of Reach, Reset: Generalization in Autonomous Visuomotor Reinforcement Learning](https://mlsanity.com/paper/2303.17600/) (2023)
- [Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculum](https://mlsanity.com/paper/2305.09943/) (2023)
- [Diffusion-Informed Probabilistic Contact Search for Multi-Finger Manipulation](https://mlsanity.com/paper/2410.00841/) (2024)
- [Hierarchical Reinforcement Learning for Articulated Tool Manipulation with Multifingered Hand](https://mlsanity.com/paper/2507.06822/) (2025)
- [Visuo-Tactile Object Pose Estimation for a Multi-Finger Robot Hand with Low-Resolution In-Hand Tactile Sensing](https://mlsanity.com/paper/2503.19893/) (2025)
- [Robustness of Robotic Manipulation: Foundations and Frontiers](https://mlsanity.com/paper/2606.31494/) (2026)
- [Distributed Dexterous Manipulation with Spatially Conditioned Multi-Agent Transformers](https://mlsanity.com/paper/2609.06930/) (2026)

## Abstract (arXiv)

Complex and contact-rich robotic manipulation tasks, particularly those that involve multi-fingered hands and underactuated object manipulation, present a significant challenge to any control method. Methods based on reinforcement learning offer an appealing choice for such settings, as they can enable robots to learn to delicately balance contact forces and dexterously reposition objects without strong modeling assumptions. However, running reinforcement learning on real-world dexterous manipulation systems often requires significant manual engineering. This negates the benefits of autonomous data collection and ease of use that reinforcement learning should in principle provide. In this paper, we describe a system for vision-based dexterous manipulation that provides a "programming-free" approach for users to define new tasks and enable robots with complex multi-fingered hands to learn to perform them through interaction. The core principle underlying our system is that, in a vision-based setting, users should be able to provide high-level intermediate supervision that circumvents challenges in teleoperation or kinesthetic teaching which allow a robot to not only learn a task efficiently but also to autonomously practice. Our system includes a framework for users to define a final task and intermediate sub-tasks with image examples, a reinforcement learning procedure that learns the task autonomously without interventions, and experimental results with a four-finger robotic hand learning multi-stage object manipulation tasks directly in the real world, without simulation, manual modeling, or reward engineering.