mlsanityData to 14 Oct 2026

ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding

arXiv 2305.08275 ↗First posted 14 May 2023cs.CVcited by 202

Not on the map. Why?

Cited by 33 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advancements in multimodal pre-training have shown promising efficacy in 3D representation learning by aligning multimodal features across 3D shapes, their 2D counterparts, and language descriptions. However, the methods used by existing frameworks to curate such multimodal data, in particular language descriptions for 3D shapes, are not scalable, and the collected language descriptions are not diverse. To address this, we introduce ULIP-2, a simple yet effective tri-modal pre-training framework that leverages large multimodal models to automatically generate holistic language descriptions for 3D shapes. It only needs 3D data as input, eliminating the need for any manual 3D annotations, and is therefore scalable to large datasets. ULIP-2 is also equipped with scaled-up backbones for better multimodal representation learning. We conduct experiments on two large-scale 3D datasets, Objaverse and ShapeNet, and augment them with tri-modal datasets of 3D point clouds, images, and language for training ULIP-2. Experiments show that ULIP-2 demonstrates substantial benefits in three downstream tasks: zero-shot 3D classification, standard 3D classification with fine-tuning, and 3D captioning (3D-to-language generation). It achieves a new SOTA of 50.6% (top-1) on Objaverse-LVIS and 84.7% (top-1) on ModelNet40 in zero-shot classification. In the ScanObjectNN benchmark for standard fine-tuning, ULIP-2 reaches an overall accuracy of 91.5% with a compact model of only 1.4 million parameters. ULIP-2 sheds light on a new paradigm for scalable multimodal 3D representation learning without human annotations and shows significant improvements over existing baselines. The code and datasets are released at https://github.com/salesforce/ULIP.

Key facts. Cited by 202 papers in the map (data through 14 Oct 2026). Builds on 53 of its 70 references that are in the map. First posted on arXiv 14 May 2023.

Builds on

53 of 70 references.

References · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
38 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  4. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  6. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  7. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 891
  8. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  9. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  11. 2019RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Cloudscited by 585
  12. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  13. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  14. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  15. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  16. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  17. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 436
  18. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  19. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 379
  20. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  21. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 264
  22. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  23. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  24. 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
  25. 2021Group-Free 3D Object Detection via Transformerscited by 215
  26. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 200
  27. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  28. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  29. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  30. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 160
  31. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 144
  32. 2020MVTN: Multi-View Transformation Network for 3D Shape Recognitioncited by 140
  33. 2021LiDAR R-CNN: An Efficient and Universal 3D Object Detectorcited by 129
  34. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 122
  35. 2018Neural Baby Talkcited by 107
  36. 2019DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processingcited by 104
  37. 2022Surface Representation for Point Cloudscited by 84
  38. 2018Grounded Video Descriptioncited by 78

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 29; 2024: 53; 2025: 67; 2026: 53.

202 papers.

Sort:

By area:

  1. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  2. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
  3. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  4. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  5. 2024Transolver: A Fast Transformer Solver for PDEs on General Geometriescited by 326
  6. 2023Text-to-3D using Gaussian Splattingcited by 299
  7. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
  8. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  9. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  10. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  11. 2025Native and Compact Structured Latents for 3D Generationcited by 175
  12. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 145
38 more
  1. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  2. 2025Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attentioncited by 110
  3. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 103
  4. 2023Human Motion Generation: A Surveycited by 96
  5. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  6. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  7. 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priorscited by 74
  8. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  9. 2024CAD-Recode: Reverse Engineering CAD Code from Point Cloudscited by 66
  10. 2024Mamba3D: Enhancing Local Features for 3D Point Cloud Analysis via State Space Modelcited by 57
  11. 2023PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulationcited by 56
  12. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 51
  13. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 49
  14. 2025Efficient Part-level 3D Object Generation via Dual Volume Packingcited by 48
  15. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  16. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 39
  17. 2023Connecting Multi-modal Contrastive Representationscited by 36
  18. 2024InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perceptioncited by 35
  19. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understandingcited by 32
  20. 2025MetaScenes: Towards Automated Replica Creation for Real-world 3D Scanscited by 31
  21. 20253D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worldscited by 30
  22. 2023ViT-Lens: Towards Omni-modal Representationscited by 29
  23. 2024SemGrasp: Semantic Grasp Generation via Language Aligned Discretizationcited by 29
  24. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26
  25. 2023Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusioncited by 24
  26. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 23
  27. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  28. 2023Extending Multi-modal Contrastive Representationscited by 21
  29. 2023A Survey of Label-Efficient Deep Learning for 3D Point Cloudscited by 20
  30. 2025Seed3D 1.0: From Images to High-Fidelity Simulation-Ready 3D Assetscited by 20
  31. 2024FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusioncited by 19
  32. 2023Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-trainingcited by 18
  33. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  34. 2024SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Mannerscited by 18
  35. 2024CLIPose: Category-Level Object Pose Estimation with Pre-trained Vision-Language Knowledgecited by 16
  36. 2024Parameter-efficient Prompt Learning for 3D Point Cloud Understandingcited by 16
  37. 2024Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understandingcited by 16
  38. 20233DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4Vcited by 15

50 most cited of 202 All 202 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding
  2. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  3. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding
  4. 2026Hunyuan3D-Buffalo 1.0: A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editing
  5. 2025Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding
  6. 2025PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Models
  7. 2025Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models
  8. 2026GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models