mlsanityData to 14 Oct 2026

Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models

arXiv 2407.13642 ↗First posted 18 Jul 2024cs.CVcited by 12

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

In this paper, we investigate the use of diffusion models which are pre-trained on large-scale image-caption pairs for open-vocabulary 3D semantic understanding. We propose a novel method, namely Diff2Scene, which leverages frozen representations from text-image generative models, along with salient-aware and geometric-aware masks, for open-vocabulary 3D semantic segmentation and visual grounding tasks. Diff2Scene gets rid of any labeled 3D data and effectively identifies objects, appearances, materials, locations and their compositions in 3D scenes. We show that it outperforms competitive baselines and achieves significant improvements over state-of-the-art methods. In particular, Diff2Scene improves the state-of-the-art method on ScanNet200 by 12%.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 81 of its 91 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

81 of 91 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter6 Sep
  2. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models8 Mar
  3. 2023Open-vocabulary Object Segmentation with Diffusion Models12 Jan
  4. 2021Label-Efficient Semantic Segmentation with Diffusion Models6 Dec

Other references · 77

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  11. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
38 more
  1. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  2. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  3. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  4. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  5. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  6. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  7. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  8. 2022Language-driven Semantic Segmentationcited by 717
  9. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  10. 2021Extract Free Dense Labels from CLIPcited by 583
  11. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  12. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  13. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  14. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  15. 2023Emergent Correspondence from Image Diffusioncited by 446
  16. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  17. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 385
  18. 2022Visual Language Maps for Robot Navigationcited by 365
  19. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  20. 2022Diffusion Models already have a Semantic Latent Spacecited by 328
  21. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  22. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  23. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  24. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  25. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  26. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  27. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  28. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  29. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  30. 2017SEGCloud: Semantic Segmentation of 3D Point Cloudscited by 179
  31. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  32. 2018Tangent Convolutions for Dense Prediction in 3Dcited by 173
  33. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  34. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  35. 2018Point Convolutional Neural Networks by Extension Operatorscited by 154
  36. 2023DDP: Diffusion Model for Dense Visual Predictioncited by 148
  37. 2023PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustmentcited by 147
  38. 2022Multi-View Transformer for 3D Visual Groundingcited by 140

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025SAS: Segment Any 3D Scene with Integrated 2D Priorscited by 6
  2. 2025PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculumcited by 4
  3. 2025CitySeg: A 3D Open Vocabulary Semantic Segmentation Foundation Model in City-scale Scenarioscited by 2
  4. 2025Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Modelcited by 2
  5. 2026GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentationcited by 1
  6. 2026Unlocking Diffusion Hierarchies: Adaptive Timestep Selection for Zero-Shot Segmentationcited by 1
  7. 2026Depth-Only Open-Vocabulary 3D Semantic Segmentation For Privacy-Preserving Robotic Applicationscited by 1
  8. 2026Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoningcited by 1
  9. 2025Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understandingcited by 0
  10. 2025OpenHuman4D: Open-Vocabulary 4D Human Parsingcited by 0
  11. 2026ClickSeg3D: Few-Click Interactive Segmentation via Semantic Embeddingscited by 0
  12. 2026Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MaskDiffusion: Exploiting Pre-trained Diffusion Models for Semantic Segmentation
  2. 2024XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation
  3. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models
  4. 2024Diffusion Features to Bridge Domain Gap for Semantic Segmentation
  5. 2025CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3D
  6. 2025Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers
  7. 2025Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation
  8. 2026MAVISEG: Manifold Propagation and Visual Prototypes for Zero-Shot Open-Vocabulary Segmentation in Diffusion Transformers