# Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network

arXiv: https://arxiv.org/abs/1809.06268 · HTML: https://mlsanity.com/paper/1809.06268/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2018.

## Built on by (most cited first, 19 of 19)

- [DexMV: Imitation Learning for Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2108.05877/) (2021)
- [AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System](https://mlsanity.com/paper/2307.04577/) (2023)
- [VideoDex: Learning Dexterity from Internet Videos](https://mlsanity.com/paper/2212.04498/) (2022)
- [Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube](https://mlsanity.com/paper/2202.10448/) (2022)
- [From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation](https://mlsanity.com/paper/2204.12490/) (2022)
- [Holo-Dex: Teaching Dexterity with Immersive Mixed Reality](https://mlsanity.com/paper/2210.06463/) (2022)
- [ACE: A Cross-Platform Visual-Exoskeletons System for Low-Cost Dexterous Teleoperation](https://mlsanity.com/paper/2408.11805/) (2024)
- [BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activities](https://mlsanity.com/paper/2503.05652/) (2025)
- [Cross-Embodiment Dexterous Grasping with Reinforcement Learning](https://mlsanity.com/paper/2410.02479/) (2024)
- [Dexterous Manipulation through Imitation Learning: A Survey](https://mlsanity.com/paper/2504.03515/) (2025)
- [ContactArt: Learning 3D Interaction Priors for Category-level Articulated Object and Hand Poses Estimation](https://mlsanity.com/paper/2305.01618/) (2023)
- [Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition](https://mlsanity.com/paper/2407.00299/) (2024)
- [Dexora: Open-source VLA for High-DoF Bimanual Dexterity](https://mlsanity.com/paper/2605.18722/) (2026)
- [Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA](https://mlsanity.com/paper/2603.08122/) (2026)
- [Single RGB-D Camera Teleoperation for General Robotic Manipulation](https://mlsanity.com/paper/2106.14396/) (2021)
- [UniFucGrasp: Human-Hand-Inspired Unified Functional Grasp Annotation Strategy and Dataset for Diverse Dexterous Hands](https://mlsanity.com/paper/2508.03339/) (2025)
- [SynHLMA:Synthesizing Hand Language Manipulation for Articulated Object with Discrete Human Object Interaction Representation](https://mlsanity.com/paper/2510.25268/) (2025)
- [Cascaded Diffusion Framework for Probabilistic Coarse-to-Fine Hand Pose Estimation](https://mlsanity.com/paper/2510.00527/) (2025)
- [Touch2Robot: Robot Touch in the Human Demonstration Loop](https://mlsanity.com/paper/2609.24660/) (2026)

## Abstract (arXiv)

In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands. Robot joint angles are directly generated from depth images of the human hand that produce visually similar robot hand poses in an end-to-end fashion. The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and simulated depth images of a robotic hand. The final training set includes 400K pairwise depth images and joint angles of a Shadow C6 robotic hand. The network evaluation results verify the superiority of TeachNet, especially regarding the high-precision condition. Imitation experiments and grasp tasks teleoperated by novice users demonstrate that TeachNet is more reliable and faster than the state-of-the-art vision-based teleoperation method.