mlsanityData to 7 Oct 2026

Monocular 3D Object Position Estimation with VLMs for Human-Robot Interaction

arXiv 2603.01224 ↗First posted 1 Mar 2026cs.CV cs.AI cs.HCcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Pre-trained general-purpose Vision-Language Models (VLM) hold the potential to enhance intuitive human-machine interactions due to their rich world knowledge and 2D object detection capabilities. However, VLMs for 3D coordinates detection tasks are rare. In this work, we investigate interactive abilities of VLMs by returning 3D object positions given a monocular RGB image from a wrist-mounted camera, natural language input, and robot states. We collected and curated a heterogeneous dataset of more than 100,000 images and finetuned a VLM using QLoRA with a custom regression head. By implementing conditional routing, our model maintains its ability to process general visual queries while adding specialized 3D position estimation capabilities. Our results demonstrate robust predictive performance with a median MAE of 13 mm on the test set and a five-fold improvement over a simpler baseline without finetuning. In about 25% of the cases, predictions are within a range considered acceptable for the robot to interact with objects.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 16 references that are in the map. First posted on arXiv 1 Mar 2026.

Builds on

16 of 16 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  3. 2023Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors10 Mar

Other references · 13

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2023LERF: Language Embedded Radiance Fieldscited by 538
  5. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  6. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  7. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 212
  8. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  9. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  10. 2022MonoDETR: Depth-guided Transformer for Monocular 3D Object Detectioncited by 85
  11. 2024Lifelong LERF: Local 3D Semantic Inventory Monitoring Using FogROS2cited by 5
  12. 2024Vision Foundation Model Driven Foreground-Aware Pseudo-LiDAR Generation for Monocular 3D Object Detectioncited by 2
1 more
  1. 2024Bringing the RT-1-X Foundation Model to a SCARA robotcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A3VLM: Actionable Articulation-Aware Vision Language Model
  2. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding
  3. 2025BOP-ASK: Object-Interaction Reasoning for Vision-Language Models
  4. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight
  5. 2025Evo-0: Vision-Language-Action Model with Implicit Spatial Understanding
  6. 20253D-Grounded Vision-Language Framework for Robotic Task Planning: Automated Prompt Synthesis and Supervised Reasoning
  7. 2026VLM3: Vision Language Models Are Native 3D Learners
  8. 2026SAM3D-Guided Object-Centric Representation Alignment for Vision-Language-Action Models