mlsanityData to 7 Oct 2026

SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding

arXiv 2401.09340 ↗First posted 17 Jan 2024cs.CV cs.AI cs.CLcited by 130

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

3D vision-language grounding, which focuses on aligning language with the 3D physical environment, stands as a cornerstone in the development of embodied agents. In comparison to recent advancements in the 2D domain, grounding language in 3D scenes faces several significant challenges: (i) the inherent complexity of 3D scenes due to the diverse object configurations, their rich attributes, and intricate relationships; (ii) the scarcity of paired 3D vision-language data to support grounded learning; and (iii) the absence of a unified learning framework to distill knowledge from grounded 3D data. In this work, we aim to address these three major challenges in 3D vision-language by examining the potential of systematically upscaling 3D vision-language learning in indoor environments. We introduce the first million-scale 3D vision-language dataset, SceneVerse, encompassing about 68K 3D indoor scenes and comprising 2.5M vision-language pairs derived from both human annotations and our scalable scene-graph-based generation approach. We demonstrate that this scaling allows for a unified pre-training framework, Grounded Pre-training for Scenes (GPS), for 3D vision-language learning. Through extensive experiments, we showcase the effectiveness of GPS by achieving state-of-the-art performance on all existing 3D visual grounding benchmarks. The vast potential of SceneVerse and GPS is unveiled through zero-shot transfer experiments in the challenging 3D vision-language tasks. Project website: https://scene-verse.github.io.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 97 of its 114 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

97 of 114 references.

References · 97

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  4. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2021Grounded Language-Image Pre-trainingcited by 1,147
  12. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  13. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  14. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  15. 2022Language-driven Semantic Segmentationcited by 711
  16. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  17. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  18. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  19. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  20. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  21. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  22. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  23. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  24. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  25. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  26. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  27. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  28. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  29. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 324
  30. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  31. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  32. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  33. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  34. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  35. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  36. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  37. 2019Structured3D: A Large Photo-realistic Dataset for Structured 3D Modelingcited by 285
  38. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280

50 most cited of 97 All 97 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 67; 2026: 32.

130 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  2. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  3. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  4. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 116
  5. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  6. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  7. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 100
  8. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  9. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  10. 2025ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splattingcited by 67
  11. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  12. 2024SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splattingcited by 63
38 more
  1. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 63
  2. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  3. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 52
  4. 2025SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretrainingcited by 51
  5. 2024Multi-modal Situated Reasoning in 3D Scenescited by 50
  6. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  7. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  8. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  9. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  10. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  11. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferencescited by 36
  12. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  13. 2024GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Groundingcited by 26
  14. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  15. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  16. 2025OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understandingcited by 24
  17. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  18. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  19. 2024UniDet3D: Multi-dataset Indoor 3D Object Detectioncited by 20
  20. 2025Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentationcited by 20
  21. 2025Describe Anything Anywhere At Any Momentcited by 19
  22. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  23. 2025InternScenes: A Large-scale Simulatable Indoor Scene Dataset with Realistic Layoutscited by 18
  24. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
  25. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMscited by 16
  26. 2025DC-Scene: Data-Centric Learning for 3D Scene Understandingcited by 15
  27. 2025Trace3D: Consistent Segmentation Lifting via Gaussian Instance Tracingcited by 13
  28. 2025CrossOver: 3D Scene Cross-Modal Alignmentcited by 12
  29. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  30. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answeringcited by 11
  31. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenescited by 11
  32. 2024Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modelingcited by 10
  33. 2025SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Modelscited by 10
  34. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  35. 2025G4Splat: Geometry-Guided Gaussian Splatting with Generative Priorcited by 10
  36. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9
  37. 2024Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Descriptioncited by 9
  38. 2024MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenescited by 9

50 most cited of 130 All 130 (JSON)