mlsanityData to 14 Oct 2026

Toward Comprehensive 3D Grounding: Orientation Grounding through Vision-Language Models

arXiv 2609.33109 ↗First posted 27 Sep 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Grounding is a core capability of spatial vision-language models, yet most existing work focuses only on where a referred object is. Many 3D tasks also require knowing how it is oriented. Although existing 3D VLMs may predict oriented boxes, box pose does not explicitly capture object-centric orientation or symmetry-induced ambiguities. We introduce orientation grounding, a referring grounding task that predicts an object's 6D orientation and axial symmetry from a language or box query in single-view or multi-view scenes. To support this task, we construct ReferOri, with 331K multi-view and 387K single-view orientation-grounding queries obtained through scalable reconstruction, consistency checking, and human verification. We further present OG-VLM, which adapts a 3D VLM with structured box/orientation outputs, sign and symmetry tokens, and geometry-aware auxiliary losses. Across single-view and multi-view benchmarks, OG-VLM substantially outperforms orientation-aware VLM baselines and surpasses object-level orientation foundation models on scene-level referring benchmarks, showing that explicit orientation grounding is a distinct and learnable capability beyond localization. Downstream results validate its benefit for orientation-related spatial reasoning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 50 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

43 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors30 May
  2. 2024Multi-modal Situated Reasoning in 3D Scenes4 Sep

Other references · 41

  1. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  3. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  4. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  6. 2016Modeling Context in Referring Expressionscited by 1,063
  7. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  8. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  9. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  11. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
  12. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
29 more
  1. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 326
  2. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  3. 2025MapAnything: Universal Feed-Forward Metric 3D Reconstructioncited by 314
  4. 2025SAM 3D: 3Dfy Anything in Imagescited by 247
  5. 2022OnePose: One-Shot Object Pose Estimation without CAD Modelscited by 157
  6. 2020Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotationscited by 155
  7. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 147
  8. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  9. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  10. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 139
  11. 2023OnePose++: Keypoint-Free One-Shot Object Pose Estimation without CAD Modelscited by 115
  12. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  13. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  14. 2025VGR: Visual Grounded Reasoningcited by 73
  15. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 68
  16. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 67
  17. 2024Omni6DPose: A Benchmark and Model for Universal 6D Object Pose Estimation and Trackingcited by 50
  18. 2023GenPose: Generative Category-level Object Pose Estimation via Diffusion Modelscited by 48
  19. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  20. 2024Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Modelscited by 45
  21. 2023POPE: 6-DoF Promptable Pose Estimation of Any Object, in Any Scene, with One Referencecited by 30
  22. 2024ImageNet3D: Towards General-Purpose Object-Level 3D Understandingcited by 25
  23. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
  24. 2026Orient Anything V2: Unifying Orientation and Rotation Understandingcited by 23
  25. 2023ZeroPose: CAD-Prompted Zero-shot Object 6D Pose Estimation in Cluttered Scenescited by 20
  26. 2025Reasoning in Space via Grounding in the Worldcited by 19
  27. 2025One2Any: One-Reference 6D Pose Estimation for Any Objectcited by 18
  28. 2025SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Graspingcited by 1
  29. 20263D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understandingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes
  2. 2026Grounded 3D-Aware Spatial Vision-Language Modeling
  3. 2026GuideGround: VLM-guided Semantic Understanding and Viewpoint-aware Reasoning for 3D Visual Grounding
  4. 2025Zero-Shot 3D Visual Grounding from Vision-Language Models
  5. 2026UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing
  6. 2026Groundbench: Multi-Resolution Polygon Grounding Exposes the Geometry Gap in Vision-Language Models
  7. 2026Scaling Diverse Language Generation for 3D Visual Grounding
  8. 2025OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding