mlsanityData to 14 Oct 2026

3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints

arXiv 2609.08224 ↗First posted 8 Sep 2026cs.RO cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Intermediate representations are key to bridging the modality gap between generalizable manipulation policies and large-scale pretrained vision-language models (VLMs). Among these, trajectory-based representations compactly represent motion-relevant cues, yet most existing approaches predict trajectories in 2D image space, resulting in intrinsic 3D ambiguity. Moreover, using 2D trajectories with depth still leaves the free-space waypoints ambiguous, limiting reliable 3D reasoning. To address this, we propose predicting 3D consistent waypoints (3DWay) from multi-view images. By reformulating 3D waypoints prediction as generating multi-view consistent 2D waypoints followed by geometric triangulation, we enable explicit 3D motion specification while preserving the strong priors of pretrained VLMs. The predicted waypoints can guide existing VLA models for better generalization or be directly executed on simple tasks. Extensive experiments show that 3DWay substantially improves 3D spatial grounding and vision-language reasoning, demonstrating strong potential for generalizable robot manipulation. Codes will be released at https://github.com/ziqin-h/3DWay.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 71 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

58 of 71 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting26 Sep
  2. 2025Igniting VLMs toward the Embodied Space15 Sep
  3. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models9 Jun
  4. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better29 May
  5. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation13 May
  6. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success27 Feb
  7. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  8. 2024A3VLM: Actionable Articulation-Aware Vision Language Model11 Jun
  9. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation24 Dec
  10. 2023Open-World Object Manipulation using Pre-trained Vision-Language Models2 Mar

Other references · 48

  1. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  2. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  3. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  4. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  5. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  6. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  7. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  8. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  9. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  10. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  11. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  12. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
36 more
  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  2. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 225
  3. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  4. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 183
  5. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 181
  6. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 175
  7. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 148
  8. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 138
  9. 2023Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulationcited by 138
  10. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 133
  11. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
  12. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 124
  13. 2024Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphscited by 119
  14. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 113
  15. 2020Visual Imitation Made Easycited by 111
  16. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 71
  17. 2025GeoVLA: Empowering 3D Representations in Vision-Language-Action Modelscited by 70
  18. 2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representationscited by 61
  19. 2025Evo-0: Vision-Language-Action Model with Implicit Spatial Understandingcited by 58
  20. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 51
  21. 2025Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulationcited by 47
  22. 2025A0: An Affordance-Aware Hierarchical Model for General Robotic Manipulationcited by 45
  23. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 43
  24. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  25. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 38
  26. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 32
  27. 2024RoboNurse-VLA: Robotic Scrub Nurse System based on Vision-Language-Action Modelcited by 29
  28. 2022K-VIL: Keypoints-based Visual Imitation Learningcited by 27
  29. 2024Tra-MoE: Learning Trajectory Prediction Model from Multiple Domains for Adaptive Policy Conditioningcited by 19
  30. 2024AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 14
  31. 2025PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policiescited by 12
  32. 2024ManiFoundation Model for General-Purpose Robotic Manipulation of Contact Synthesis with Arbitrary Objects and Robotscited by 11
  33. 2024VIEW: Visual Imitation Learning with Waypointscited by 10
  34. 2024Waypoint-Based Reinforcement Learning for Robot Manipulation Taskscited by 9
  35. 2025Sketch-to-Skill: Bootstrapping Robot Learning with Human Drawn Trajectory Sketchescited by 9
  36. 2024Sketch-MoMa: Teleoperation for Mobile Manipulator via Interpretation of Hand-Drawn Sketchescited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Bridge3D: Enabling Vision-Language-Action Models to See and Act in 3D
  2. 2026Robotic Manipulation is Vision-to-Geometry Mapping: Vision-Geometry Backbones over Language and Video Models
  3. 2025GeoPredict: Leveraging Predictive Kinematics and 3D Gaussian Geometry for Precise VLA Manipulation
  4. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policy
  5. 2026VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation
  6. 20253D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Tasks
  7. 2026Dexterity-BEV: Aligning 3D World and Actions for Generalizable Robot Policies Learning
  8. 2026StrataVLA: Hierarchical and Efficient 3D Geometric Grounding for Vision-Language-Action Models