mlsanityData to 14 Oct 2026

Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models

arXiv 2207.11514 ↗First posted 23 Jul 2022cs.CV cs.ROcited by 95

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an open-set vocabulary and out-of-domain visual inputs - a critical skill for robots to operate in the unstructured 3D world. Towards this end, we propose Semantic Abstraction (SemAbs), a framework that equips 2D Vision-Language Models (VLMs) with new 3D spatial capabilities, while maintaining their zero-shot robustness. We achieve this abstraction using relevancy maps extracted from CLIP, and learn 3D spatial and geometric reasoning skills on top of those abstractions in a semantic-agnostic manner. We demonstrate the usefulness of SemAbs on two open-world 3D scene understanding tasks: 1) completing partially observed objects and 2) localizing hidden objects from language descriptions. Experiments show that SemAbs can generalize to novel vocabulary, materials/lighting, classes, and domains (i.e., real-world scans) from training on limited 3D synthetic data. Code and data is available at https://semantic-abstraction.cs.columbia.edu/

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 62 references that are in the map. First posted on arXiv 23 Jul 2022.

Builds on

46 of 62 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  8. 2017Decoupled Weight Decay Regularizationcited by 1,831
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  10. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  12. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
34 more
  1. 2021Robust fine-tuning of zero-shot modelscited by 797
  2. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 658
  3. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  4. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  5. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  6. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  7. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  8. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  11. 2016Semantic Scene Completion from a Single Depth Imagecited by 472
  12. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  13. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  14. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  15. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  16. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 217
  17. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  18. 2016Learning Shape Abstractions by Assembling Volumetric Primitivescited by 181
  19. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  20. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 120
  21. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  22. 2015Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Imagescited by 111
  23. 2017ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scanscited by 103
  24. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 89
  25. 2021The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuningcited by 75
  26. 2021Learning to Compose Visual Relationscited by 72
  27. 2021Semantic Scene Completion via Integrating Instances and Scene in-the-Loopcited by 55
  28. 2020SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigationcited by 44
  29. 2019Cascaded Context Pyramid for Full-Resolution 3D Semantic Scene Completioncited by 43
  30. 2022Rethinking Attention-Model Explainability through Faithfulness Violation Testcited by 43
  31. 2022Optimizing Relevance Maps of Vision Transformers Improves Robustnesscited by 41
  32. 2021Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examplescited by 24
  33. 2022Adapting CLIP For Phrase Localization Without Further Trainingcited by 21
  34. 2015Predicting Complete 3D Models of Indoor Scenescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 28; 2024: 32; 2025: 21; 2026: 9.

95 papers.

Sort:

By area:

  1. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  2. 2023LERF: Language Embedded Radiance Fieldscited by 543
  3. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  4. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  5. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  6. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  7. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  8. 2023Segment Any 3D Gaussianscited by 161
  9. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  10. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  11. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  12. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
38 more
  1. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
  2. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  3. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  4. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  5. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  6. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 82
  7. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 80
  8. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  9. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 71
  10. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  11. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  12. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  13. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 69
  14. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64
  15. 2023Segment Anything in 3D with Radiance Fieldscited by 62
  16. 2023Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representationcited by 54
  17. 2023FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Modelscited by 47
  18. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  19. 2023Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Drivingcited by 37
  20. 20233D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillationcited by 35
  21. 2023GrowSP: Unsupervised Semantic Segmentation of 3D Point Cloudscited by 34
  22. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Roboticscited by 34
  23. 2024N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fieldscited by 34
  24. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  25. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulationcited by 32
  26. 2024TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understandingcited by 27
  27. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25
  28. 2023OVO: Open-Vocabulary Occupancycited by 24
  29. 2023Grounding Classical Task Planners via Vision-Language Modelscited by 23
  30. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 23
  31. 2024Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentationcited by 20
  32. 2024SLGaussian: Fast Language Gaussian Splatting in Sparse Viewscited by 18
  33. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  34. 2024Occam's LGS: An Efficient Approach for Language Gaussian Splattingcited by 16
  35. 2024RelationField: Relate Anything in Radiance Fieldscited by 16
  36. 2023Language-EXtended Indoor SLAM (LEXIS): A Versatile System for Real-time Visual Scene Understandingcited by 15
  37. 2024DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planningcited by 14
  38. 20243D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillationcited by 13

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
  2. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding
  3. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes
  4. 2025Queryable 3D Scene Representation: A Multi-Modal Framework for Semantic Reasoning and Robotic Task Planning
  5. 2025Text-Scene: A Scene-to-Language Parsing Framework for 3D Scene Understanding
  6. 2026A Scene Graph Backed Approach to Open Set Semantic Mapping
  7. 2025ZING-3D: Zero-shot Incremental 3D Scene Graphs via Vision-Language Models
  8. 2026Relationship-Aware Hierarchical 3D Scene Graph for Task Reasoning