# From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes

arXiv: https://arxiv.org/abs/2506.04897 · HTML: https://mlsanity.com/paper/2506.04897/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 75 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 30 of 48)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [An Embodied Generalist Agent in 3D World](https://mlsanity.com/paper/2311.12871/) (2023)
- [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021)
- [PointLLM: Empowering Large Language Models to Understand Point Clouds](https://mlsanity.com/paper/2308.16911/) (2023)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding](https://mlsanity.com/paper/2403.11481/) (2024)
- [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020)
- [LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning](https://mlsanity.com/paper/2311.18651/) (2023)
- [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023)
- [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019)
- [Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers](https://mlsanity.com/paper/2312.08168/) (2023)
- [Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning](https://mlsanity.com/paper/2403.11401/) (2024)
- [Multi3DRefer: Grounding Text Description to Multiple 3D Objects](https://mlsanity.com/paper/2309.05251/) (2023)
- [LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent](https://mlsanity.com/paper/2309.12311/) (2023)
- [SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding](https://mlsanity.com/paper/2401.09340/) (2024)
- [Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds](https://mlsanity.com/paper/2112.08879/) (2021)
- [Language Conditioned Spatial Relation Reasoning for 3D Object Grounding](https://mlsanity.com/paper/2211.09646/) (2022)
- [Grounded 3D-LLM with Referent Tokens](https://mlsanity.com/paper/2405.10370/) (2024)
- [3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection](https://mlsanity.com/paper/2204.06272/) (2022)

## Built on by (most cited first, 4 of 4)

- [LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation](https://mlsanity.com/paper/2604.11789/) (2026)
- [Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenes](https://mlsanity.com/paper/2602.00593/) (2026)
- [SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessity](https://mlsanity.com/paper/2606.25634/) (2026)
- [Hard Vision, Easy Vision: What GPT-6 Astra Reveals Across Computer Vision](https://mlsanity.com/paper/2609.35718/) (2026)

## Abstract (arXiv)

3D visual grounding has made notable progress in localizing objects within complex 3D scenes. However, grounding referring expressions beyond objects in 3D scenes remains unexplored. In this paper, we introduce Anywhere3D-Bench, a holistic 3D visual grounding benchmark consisting of 2,886 referring expression-3D bounding box pairs spanning four different grounding levels: human-activity areas, unoccupied space beyond objects, individual objects in the scene, and fine-grained object parts. We assess a range of state-of-the-art 3D visual grounding methods alongside large language models (LLMs) and multimodal LLMs (MLLMs) on Anywhere3D-Bench. Experimental results reveal that space-level and part-level visual grounding pose the greatest challenges: space-level tasks require a more comprehensive spatial reasoning ability, for example, modeling distances and spatial relations within 3D space, while part-level tasks demand fine-grained perception of object composition. Even the best-performing models, Google Gemini-2.5-Pro and OpenAI o3, achieve just around 30% accuracy on space-level tasks and around 40% on part-level tasks, significantly lower than its performance on area-level and object-level tasks. These findings underscore a critical gap in current models' capacity to understand and reason about 3D scenes beyond object-level semantics.