# ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Grounding

arXiv: https://arxiv.org/abs/2501.01366 · HTML: https://mlsanity.com/paper/2501.01366/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 71 references that are in the map.
- First posted on arXiv 2 Jan 2025.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [An Embodied Generalist Agent in 3D World](https://mlsanity.com/paper/2311.12871/) (2023)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario](https://mlsanity.com/paper/2305.14836/) (2023)
- [Multi3DRefer: Grounding Text Description to Multiple 3D Objects](https://mlsanity.com/paper/2309.05251/) (2023)
- [LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent](https://mlsanity.com/paper/2309.12311/) (2023)
- [SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding](https://mlsanity.com/paper/2401.09340/) (2024)
- [Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds](https://mlsanity.com/paper/2112.08879/) (2021)
- [Language Conditioned Spatial Relation Reasoning for 3D Object Grounding](https://mlsanity.com/paper/2211.09646/) (2022)
- [Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories](https://mlsanity.com/paper/2306.09224/) (2023)
- [EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding](https://mlsanity.com/paper/2209.14941/) (2022)
- [PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks](https://mlsanity.com/paper/2411.00081/) (2024)
- [Unifying 3D Vision-Language Understanding via Promptable Queries](https://mlsanity.com/paper/2405.11442/) (2024)
- [UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding](https://mlsanity.com/paper/2212.00836/) (2022)
- [Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding](https://mlsanity.com/paper/2311.15383/) (2023)
- [ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilities](https://mlsanity.com/paper/2407.01525/) (2024)
- [MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations](https://mlsanity.com/paper/2406.09401/) (2024)
- [3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination](https://mlsanity.com/paper/2406.05132/) (2024)
- [ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenes](https://mlsanity.com/paper/2212.06250/) (2022)
- [M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts](https://mlsanity.com/paper/2312.10763/) (2023)
- [VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation](https://mlsanity.com/paper/2411.03540/) (2024)

## Built on by (most cited first, 7 of 7)

- [OpenLex3D: A Tiered Evaluation Benchmark for Open-Vocabulary 3D Scene Representations](https://mlsanity.com/paper/2503.19764/) (2025)
- [MLFM: Multi-Layered Feature Maps for Richer Language Understanding in Zero-Shot Semantic Navigation](https://mlsanity.com/paper/2507.07299/) (2025)
- [ChangingGrounding: 3D Visual Grounding in Changing Scenes](https://mlsanity.com/paper/2510.14965/) (2025)
- [SHREC 2025: Retrieval of Optimal Objects for Multi-modal Enhanced Language and Spatial Assistance (ROOMELSA)](https://mlsanity.com/paper/2508.08781/) (2025)
- [Scaling Diverse Language Generation for 3D Visual Grounding](https://mlsanity.com/paper/2606.20946/) (2026)
- [SeqAlign3DVG: A Sequence-Aligned Benchmark and Voxel Reasoning Framework for 3D Visual Grounding](https://mlsanity.com/paper/2608.30451/) (2026)
- [Task-Adaptive Grounded 3D-Programmers Using 2D VLMs](https://mlsanity.com/paper/2610.02021/) (2026)

## Abstract (arXiv)

3D visual grounding (3DVG) involves localizing entities in a 3D scene referred to by natural language text. Such models are useful for embodied AI and scene retrieval applications, which involve searching for objects or patterns using natural language descriptions. While recent works have focused on LLM-based scaling of 3DVG datasets, these datasets do not capture the full range of potential prompts which could be specified in the English language. To ensure that we are scaling up and testing against a useful and representative set of prompts, we propose a framework for linguistically analyzing 3DVG prompts and introduce Visual Grounding with Diverse Language in 3D (ViGiL3D), a diagnostic dataset for evaluating visual grounding methods against a diverse set of language patterns. We evaluate existing open-vocabulary 3DVG methods to demonstrate that these methods are not yet proficient in understanding and identifying the targets of more challenging, out-of-distribution prompts, toward real-world applications.