mlsanityData to 7 Oct 2026

OpenScene: 3D Scene Understanding with Open Vocabularies

arXiv 2211.15654 ↗First posted 28 Nov 2022cs.CVcited by 556

Not on the map. Why?

Cited by 67 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Traditional 3D scene understanding approaches rely on labeled 3D datasets to train a model for a single task with supervision. We propose OpenScene, an alternative approach where a model predicts dense features for 3D scene points that are co-embedded with text and image pixels in CLIP feature space. This zero-shot approach enables task-agnostic training and open-vocabulary queries. For example, to perform SOTA zero-shot 3D semantic segmentation it first infers CLIP features for every 3D point and later classifies them based on similarities to embeddings of arbitrary class labels. More interestingly, it enables a suite of open-vocabulary scene understanding applications that have never been done before. For example, it allows a user to enter an arbitrary text query and then see a heat map indicating which parts of a scene match. Our approach is effective at identifying objects, materials, affordances, activities, and room types in complex 3D scenes, all using a single model trained without any labeled 3D data.

Key facts. Cited by 556 papers in the map (data through 7 Oct 2026). Builds on 57 of its 68 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

57 of 68 references.

References · 57

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  10. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  11. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  12. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
38 more
  1. 2022Language-driven Semantic Segmentationcited by 711
  2. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  3. 2021Extract Free Dense Labels from CLIPcited by 580
  4. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  5. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  6. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  7. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  8. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  9. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  11. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
  12. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  13. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  14. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  15. 2020OccuSeg: Occupancy-aware 3D Instance Segmentationcited by 177
  16. 2018Tangent Convolutions for Dense Prediction in 3Dcited by 173
  17. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  18. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  19. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  20. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 134
  21. 2021Mix3D: Out-of-Context Data Augmentation for 3D Scenescited by 126
  22. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  23. 2021MSeg: A Composite Dataset for Multi-domain Semantic Segmentationcited by 122
  24. 2017ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scanscited by 103
  25. 20213D AffordanceNet: A Benchmark for Visual Object Affordance Understandingcited by 100
  26. 2020Virtual Multi-view Fusion for 3D Semantic Segmentationcited by 99
  27. 2021Bidirectional Projection Network for Cross Dimension Scene Understandingcited by 95
  28. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  29. 2022LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perceptioncited by 74
  30. 2019Putting Humans in a Scene: Learning Affordance in 3D Indoor Environmentscited by 72
  31. 2021Learning 3D Semantic Segmentation with only 2D Image Supervisioncited by 60
  32. 2022Panoptic-PHNet: Towards Real-Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmapcited by 59
  33. 2021Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Cloudscited by 55
  34. 2022Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentationcited by 47
  35. 2018TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshescited by 45
  36. 2021Contrastive Multimodal Fusion with TupleInfoNCEcited by 43
  37. 2020DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshescited by 42
  38. 2019Transductive Zero-Shot Learning for 3D Point Cloud Classificationcited by 39

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 83; 2024: 148; 2025: 181; 2026: 144.

556 papers.

Sort:

By area:

  1. 2023LERF: Language Embedded Radiance Fieldscited by 538
  2. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 387
  3. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  4. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  5. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  6. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  7. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  8. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  9. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  10. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  11. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  12. 2023Segment Any 3D Gaussianscited by 158
38 more
  1. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  2. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 141
  3. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  4. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  5. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  6. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  7. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  8. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  9. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  10. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  11. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  12. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 94
  13. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 93
  14. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  15. 2025PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyondcited by 91
  16. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
  17. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  18. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  19. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  20. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 81
  21. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  22. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 77
  23. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  24. 2023OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learningcited by 75
  25. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
  26. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  27. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 70
  28. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 68
  29. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  30. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  31. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  32. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64
  33. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  34. 2023Segment Anything in 3D with Radiance Fieldscited by 61
  35. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  36. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  37. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  38. 2023Factor Fields: A Unified Framework for Neural Fields and Beyondcited by 58

50 most cited of 556 All 556 (JSON)