# Tele-Catch: Adaptive Teleoperation for Dexterous Dynamic 3D Object Catching

arXiv: https://arxiv.org/abs/2603.28427 · HTML: https://mlsanity.com/paper/2603.28427/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Builds on (most cited first, 28 of 28)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Open-TeleVision: Teleoperation with Immersive Active Visual Feedback](https://mlsanity.com/paper/2407.01512/) (2024)
- [DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation](https://mlsanity.com/paper/2210.02697/) (2022)
- [Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation](https://mlsanity.com/paper/2403.04436/) (2024)
- [Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning](https://mlsanity.com/paper/2206.08686/) (2022)
- [DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System](https://mlsanity.com/paper/1910.03135/) (2019)
- [ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning](https://mlsanity.com/paper/2503.21860/) (2025)
- [ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2404.15709/) (2024)
- [FP3: A 3D Foundation Policy for Robotic Manipulation](https://mlsanity.com/paper/2503.08950/) (2025)
- [DexterityGen: Foundation Controller for Unprecedented Dexterity](https://mlsanity.com/paper/2502.04307/) (2025)
- [Cross-Embodiment Dexterous Grasping with Reinforcement Learning](https://mlsanity.com/paper/2410.02479/) (2024)
- [DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics](https://mlsanity.com/paper/2304.03223/) (2023)
- [SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation](https://mlsanity.com/paper/2310.16838/) (2023)
- [DOGlove: Dexterous Manipulation with a Low-Cost Open-Source Haptic Force Feedback Glove](https://mlsanity.com/paper/2502.07730/) (2025)
- [CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-World](https://mlsanity.com/paper/2502.08449/) (2025)
- [DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References](https://mlsanity.com/paper/2502.09614/) (2025)
- [DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation](https://mlsanity.com/paper/2411.18562/) (2024)
- [Geometric Retargeting: A Principled, Ultrafast Neural Hand Retargeting Algorithm](https://mlsanity.com/paper/2503.07541/) (2025)
- [ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes](https://mlsanity.com/paper/2506.14317/) (2025)
- [Dexterous Teleoperation of 20-DoF ByteDexter Hand via Human Motion Retargeting](https://mlsanity.com/paper/2507.03227/) (2025)
- [DexCatch: Learning to Catch Arbitrary Objects with Dexterous Hands](https://mlsanity.com/paper/2310.08809/) (2023)
- [Contact-Implicit Model Predictive Control for Dexterous In-hand Manipulation: A Long-Horizon and Robust Approach](https://mlsanity.com/paper/2402.18897/) (2024)
- [Whole-Body Teleoperation for Mobile Manipulation at Zero Added Cost](https://mlsanity.com/paper/2409.15095/) (2024)
- [Sampling-Based Model Predictive Control for Dexterous Manipulation on a Biomimetic Tendon-Driven Hand](https://mlsanity.com/paper/2411.06183/) (2024)
- [Exo-ViHa: A Cross-Platform Exoskeleton System with Visual and Haptic Feedback for Efficient Dexterous Skill Learning](https://mlsanity.com/paper/2503.01543/) (2025)
- [GEX: Democratizing Dexterity with Fully-Actuated Dexterous Hand and Exoskeleton Glove](https://mlsanity.com/paper/2506.04982/) (2025)

## Abstract (arXiv)

Teleoperation is a key paradigm for transferring human dexterity to robots, yet most prior work targets objects that are initially static, such as grasping or manipulation. Dynamic object catch, where objects move before contact, remains underexplored. Pure teleoperation in this task often fails due to timing, pose, and force errors, highlighting the need for shared autonomy that combines human input with autonomous policies. To this end, we present Tele-Catch, a systematic framework for dexterous hand teleoperation in dynamic object catching. At its core, we design DAIM, a dynamics-aware adaptive integration mechanism that realizes shared autonomy by fusing glove-based teleoperation signals into the diffusion policy denoising process. It adaptively modulates control based on the interaction object state. To improve policy robustness, we introduce DP-U3R, which integrates unsupervised geometric representations from point cloud observations into diffusion policy learning, enabling geometry-aware decision making. Extensive experiments demonstrate that Tele-Catch significantly improves accuracy and robustness in dynamic catching tasks, while also exhibiting consistent gains across distinct dexterous hand embodiments and previously unseen object categories.