# ZeroDex: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning

arXiv: https://arxiv.org/abs/2606.19340 · HTML: https://mlsanity.com/paper/2606.19340/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 30 of 33)

- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model](https://mlsanity.com/paper/2501.15830/) (2025)
- [CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation](https://mlsanity.com/paper/2411.19650/) (2024)
- [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics](https://mlsanity.com/paper/2406.10721/) (2024)
- [Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets](https://mlsanity.com/paper/2109.13396/) (2021)
- [DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control](https://mlsanity.com/paper/2502.05855/) (2025)
- [PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs](https://mlsanity.com/paper/2402.07872/) (2024)
- [DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation](https://mlsanity.com/paper/2403.07788/) (2024)
- [DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation](https://mlsanity.com/paper/2210.02697/) (2022)
- [MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting](https://mlsanity.com/paper/2403.03174/) (2024)
- [Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos](https://mlsanity.com/paper/2507.15597/) (2025)
- [Dreamitate: Real-World Visuomotor Policy Learning via Video Generation](https://mlsanity.com/paper/2406.16862/) (2024)
- [Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer](https://mlsanity.com/paper/2510.03342/) (2025)
- [General Flow as Foundation Affordance for Scalable Robot Learning](https://mlsanity.com/paper/2401.11439/) (2024)
- [Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube](https://mlsanity.com/paper/2202.10448/) (2022)
- [VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding](https://mlsanity.com/paper/2410.13860/) (2024)
- [RLDX-1 Technical Report](https://mlsanity.com/paper/2605.03269/) (2026)
- [Dexonomy: Synthesizing All Dexterous Grasp Types in a Grasp Taxonomy](https://mlsanity.com/paper/2504.18829/) (2025)

## Abstract (arXiv)

We present ZeroDex, a zero-shot framework for long-horizon dexterous manipulation that grounds language instructions into executable 3D task plans from calibrated multi-view RGB images. Rather than training an end-to-end policy, our system uses a vision-language model (VLM) to produce reference-frame task grounding and primitive-level 2D keypoints, then lifts them into 3D via multi-view fusion. This lifting combines triangulation of view-wise VLM groundings with reference-view ray voting, which searches along a semantic camera ray for geometrically consistent candidates across neighboring views. The resulting 3D keypoints support both pick-and-place and tool-use: for tool-use, we retrieve an object-centric atomic action corresponding to the inferred skill category and align its stored 6D tool trajectory to the scene; for dexterous execution, we expand the lifted grasp keypoint into a task-conditioned grasp affordance region and generate feasible grasp-motion pairs with an arm-hand motion generator. Real-world experiments show improved 3D grounding accuracy and execution reliability over single-view RGB-D grounding and fine-tuned VLA baselines. We further demonstrate long-horizon manipulation through closed-loop status verification and replan, enabling zero-shot execution on unseen objects and tool-use tasks in novel scenes.