mlsanityData to 14 Oct 2026

Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding

arXiv 2504.19500 ↗First posted 28 Apr 2025cs.CV cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Open-vocabulary 3D scene understanding is pivotal for enhancing physical intelligence, as it enables embodied agents to interpret and interact dynamically within real-world environments. This paper introduces MPEC, a novel Masked Point-Entity Contrastive learning method for open-vocabulary 3D semantic segmentation that leverages both 3D entity-language alignment and point-entity consistency across different point cloud views to foster entity-specific feature representations. Our method improves semantic discrimination and enhances the differentiation of unique instances, achieving state-of-the-art results on ScanNet for open-vocabulary 3D semantic segmentation and demonstrating superior zero-shot scene understanding capabilities. Extensive fine-tuning experiments on 8 datasets, spanning from low-level perception to high-level reasoning tasks, showcase the potential of learned 3D features, driving consistent performance gains across varied 3D scene understanding tasks. Project website: https://mpec-3d.github.io/

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 84 of its 96 references that are in the map. First posted on arXiv 28 Apr 2025.

Builds on

84 of 96 references.

References · 84

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023Segment Anythingcited by 8,234
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  9. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  10. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  11. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  12. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
38 more
  1. 2022Language-driven Semantic Segmentationcited by 717
  2. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  3. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  4. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  5. 2021Extract Free Dense Labels from CLIPcited by 583
  6. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  7. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 504
  8. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  9. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  11. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  12. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  13. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 328
  14. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
  15. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  16. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  17. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  18. 2019Structured3D: A Large Photo-realistic Dataset for Structured 3D Modelingcited by 286
  19. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  20. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
  21. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 219
  22. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  23. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  24. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 216
  25. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  26. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  27. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  28. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 188
  29. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
  30. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
  31. 2021Spatio-temporal Self-Supervised Representation Learning for 3D Point Cloudscited by 180
  32. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  33. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  34. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  35. 2023SAM3D: Segment Anything in 3D Scenescited by 157
  36. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 147
  37. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  38. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2026Multiplayer Interactive World Models with Representation Autoencoderscited by 19
  2. 2025PartSAM: A Scalable Promptable Part Segmentation Model Trained on Native 3D Datacited by 16
  3. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 5
  4. 2025FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understandingcited by 2
  5. 2026What if? Emulative Simulation with World Models for Situated Reasoningcited by 2
  6. 2025GeoPurify: A Data-Efficient Geometric Distillation Framework for Open-Vocabulary 3D Segmentationcited by 1
  7. 2026GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentationcited by 1
  8. 2026PointCSP: Cross-Sample Semantic Propagation and Stability Preservation in Self-Supervised Point Cloud Learningcited by 1
  9. 2026Depth-Only Open-Vocabulary 3D Semantic Segmentation For Privacy-Preserving Robotic Applicationscited by 1
  10. 2026FoundObj: Self-supervised Foundation Models as Rewards for Label-free 3D Object Segmentationcited by 0