# VLA-3D: A Dataset for 3D Semantic Scene Understanding and Navigation

arXiv: https://arxiv.org/abs/2411.03540 · HTML: https://mlsanity.com/paper/2411.03540/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- First posted on arXiv 5 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Built on by (most cited first, 21 of 21)

- [Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challenges](https://mlsanity.com/paper/2505.04769/) (2025)
- [g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks](https://mlsanity.com/paper/2411.17030/) (2024)
- [SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Models](https://mlsanity.com/paper/2504.18684/) (2025)
- [SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes](https://mlsanity.com/paper/2507.07781/) (2025)
- [PlaceIt3D: Language-Guided Object Placement in Real 3D Scenes](https://mlsanity.com/paper/2505.05288/) (2025)
- [ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Grounding](https://mlsanity.com/paper/2501.01366/) (2025)
- [ReFineVLA: Reasoning-Aware Teacher-Guided Transfer Fine-Tuning](https://mlsanity.com/paper/2505.19080/) (2025)
- [A Review of 3D Object Detection with Vision-Language Models](https://mlsanity.com/paper/2504.18738/) (2025)
- [An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challenges](https://mlsanity.com/paper/2512.11362/) (2025)
- [SeqWalker: Sequential-Horizon Vision-and-Language Navigation with Hierarchical Planning](https://mlsanity.com/paper/2601.04699/) (2026)
- [EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environments](https://mlsanity.com/paper/2503.08604/) (2025)
- [SSR: Pushing the Limit of Spatial Intelligence with Structured Scene Reasoning](https://mlsanity.com/paper/2603.00409/) (2026)
- [CoPa-SG: Dense Scene Graphs with Parametric and Proto-Relations](https://mlsanity.com/paper/2506.21357/) (2025)
- [ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuning](https://mlsanity.com/paper/2604.17800/) (2026)
- [HOIverse: A Synthetic Scene Graph Dataset With Human Object Interactions](https://mlsanity.com/paper/2506.19639/) (2025)
- [Sceniris: A Fast Procedural Scene Generation Framework](https://mlsanity.com/paper/2512.16896/) (2025)
- [ObjChangeVR: Object State Change Reasoning from Continuous Egocentric Views in VR Environments](https://mlsanity.com/paper/2603.06648/) (2026)
- [OpenSGA: Efficient 3D Scene Graph Alignment in the Open World](https://mlsanity.com/paper/2605.10484/) (2026)
- [Scaling Diverse Language Generation for 3D Visual Grounding](https://mlsanity.com/paper/2606.20946/) (2026)
- [A Modular Vision-Language-Action Robotics Framework for Indoor Environments](https://mlsanity.com/paper/2606.31144/) (2026)
- [AnchorVLN: Geometry-Anchored Vision-Language Grounding Reasoning for Open-Vocabulary Navigation](https://mlsanity.com/paper/2609.12285/) (2026)

## Abstract (arXiv)

With the recent rise of Large Language Models (LLMs), Vision-Language Models (VLMs), and other general foundation models, there is growing potential for multimodal, multi-task embodied agents that can operate in diverse environments given only natural language as input. One such application area is indoor navigation using natural language instructions. However, despite recent progress, this problem remains challenging due to the spatial reasoning and semantic understanding required, particularly in arbitrary scenes that may contain many objects belonging to fine-grained classes. To address this challenge, we curate the largest real-world dataset for Vision and Language-guided Action in 3D Scenes (VLA-3D), consisting of over 11.5K scanned 3D indoor rooms from existing datasets, 23.5M heuristically generated semantic relations between objects, and 9.7M synthetically generated referential statements. Our dataset consists of processed 3D point clouds, semantic object and room annotations, scene graphs, navigable free space annotations, and referential language statements that specifically focus on view-independent spatial relations for disambiguating objects. The goal of these features is to aid the downstream task of navigation, especially on real-world systems where some level of robustness must be guaranteed in an open world of changing scenes and imperfect language. We benchmark our dataset with current state-of-the-art models to obtain a performance baseline. All code to generate and visualize the dataset is publicly released, see https://github.com/HaochenZ11/VLA-3D. With the release of this dataset, we hope to provide a resource for progress in semantic 3D scene understanding that is robust to changes and one which will aid the development of interactive indoor navigation systems.