mlsanityData to 7 Oct 2026

RGB-Pointmap Pretraining for Unified 3D Scene Understanding

arXiv 2604.02546 ↗First posted 2 Apr 2026cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Pretraining 3D encoders through alignment with Contrastive Language-Image Pre-training (CLIP) has emerged as a promising direction for learning generalizable representations for 3D scene understanding. In this paper, we propose UniScene3D, a transformer-based framework that learns unified 3D scene representations from multi-view RGB-Pointmap inputs by leveraging the priors of a pretrained 2D foundation model. For robust RGB-Pointmap representation learning, we introduce cross-view geometric alignment and grounded view alignment to enforce geometric and semantic consistency across views. Extensive low-shot and task-specific fine-tuning on viewpoint grounding, scene retrieval, scene classification, and 3D visual question answering demonstrates state-of-the-art performance. These results establish UniScene3D as an effective framework for unified 3D scene understanding. Project page: https://yebulabula.github.io/UniScene3D/

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 58 of its 66 references that are in the map. First posted on arXiv 2 Apr 2026.

Builds on

58 of 66 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  5. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  6. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  7. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  8. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  9. 2019BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networkscited by 487
  10. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  11. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  12. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
38 more
  1. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  2. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  3. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  4. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  5. 2019Structured3D: A Large Photo-realistic Dataset for Structured 3D Modelingcited by 285
  6. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  7. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  8. 2023Data Filtering Networkscited by 247
  9. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  10. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  11. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  12. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  13. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  14. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  15. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  16. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  17. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  18. 2017SEGCloud: Semantic Segmentation of 3D Point Cloudscited by 178
  19. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  20. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  21. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  22. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  23. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  24. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 93
  25. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 78
  26. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  27. 2025SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretrainingcited by 51
  28. 2025LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPScited by 40
  29. 2024jina-clip-v2: Multilingual Multimodal Embeddings for Text and Imagescited by 36
  30. 2021Deep Learning Based 3D Segmentation: A Surveycited by 30
  31. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
  32. 20233D-Aware Visual Question Answering about Parts, Poses and Occlusionscited by 16
  33. 2024CLIP-GS: CLIP-Informed Gaussian Splatting for View-Consistent 3D Indoor Semantic Understandingcited by 16
  34. 2024"Where am I?" Scene Retrieval with Languagecited by 15
  35. 2023Point-GCC: Universal Self-supervised 3D Scene Pre-training via Geometry-Color Contrastcited by 14
  36. 2024Duoduo CLIP: Efficient 3D Understanding with Multi-View Imagescited by 12
  37. 2025CrossOver: 3D Scene Cross-Modal Alignmentcited by 12
  38. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answeringcited by 11

50 most cited of 58 All 58 (JSON)

Built on by

Not cited yet.