Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
3D localization in Multimodal Large Language Models (MLLMs), including 3D object detection and 3D visual grounding, is fundamentally limited by camera intrinsic ambiguity: the same image admits different 3D scenes under different cameras. Existing MLLMs either ignore camera parameters and overfit to a canonical training intrinsic, or retrieve depth and 3D cues from external tools but treat the returned values as reference cues (numerical hints that the model is free to interpret implicitly), both preventing camera information from being deterministically propagated into the prediction. We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables. The proposed framework proactively retrieves camera intrinsics and samples multi-point metric depths, writes the pinhole back-projection equation hatX = (uc - cx)barZ/fx explicitly in Chain-of-Thought (CoT), and substitutes tool outputs into the formula before regressing the final 9-DoF bounding box. On both 3D object detection and 3D visual grounding tasks under rescaled camera intrinsics from 0.5× to 1.5×, our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale. Code and data will be released.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 49 references that are in the map. First posted on arXiv 19 May 2026.
Builds on
46 of 49 references.
Closest ancestors · 17
Same direction, newest first
- 2026LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models8 Apr
- 2026On the Generalization Capacities of MLLMs for Spatial Intelligence5 Mar
- 2026Think3D: Thinking with Space for Spatial Reasoning19 Jan
- 2025LAST: LeArning to Think in Space and Time for Generalist Vision-Language Models24 Nov
- 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics4 Jun
- 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors30 May
- 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence29 May
- 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction26 May
- 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning20 May
- 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning18 May
- 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning13 May
- 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D29 Mar
5 more
- 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models9 Mar
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
- 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
- 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
- 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
Other references · 29
- 2025Qwen3 Technical Reportcited by 8,820
- 2023Visual Instruction Tuningcited by 7,869
- 2024GPT-4o System Cardcited by 4,588
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2023Qwen Technical Reportcited by 3,344
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
- 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2025SAM 3: Segment Anything with Conceptscited by 906
- 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
- 2022PAL: Program-aided Language Modelscited by 652
17 more
- 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
- 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
- 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 391
- 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 278
- 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 268
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
- 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
- 2025UniDepthV2: Universal Monocular Metric Depth Estimation Made Simplercited by 173
- 2025Thyme: Think Beyond Imagescited by 167
- 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
- 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 83
- 2025Grounded Reinforcement Learning for Visual Reasoningcited by 83
- 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 82
- 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
- 2025Reinforced Visual Perception with Toolscited by 38
- 2024Neuro-Symbolic Data Generation for Math Reasoningcited by 22
- 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 12
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 20263D-Aware VLMs with Implicit and Explicit Geometriescited by 0
- 20263D-Aware VLMs with Implicit and Explicit Geometries23 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2026Seeing Is Not Measuring: Tool-Augmented Metric Spatial Reasoning for Vision-Language Models
- 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight
- 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models
- 2026Unlocking Dense Metric Depth Estimation in VLMs
- 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models
- 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding
- 2025DepthLM: Metric Depth From Vision Language Models
- 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs