mlsanityData to 14 Oct 2026

ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding

arXiv 2212.05171 ↗First posted 10 Dec 2022cs.CVcited by 267

Not on the map. Why?

Cited by 48 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recognition capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of images, texts, and 3D point clouds by pre-training with object triplets from the three modalities. To overcome the shortage of training triplets, ULIP leverages a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs. Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets. ULIP is agnostic to 3D backbone networks and can easily be integrated into any 3D architecture. Experiments show that ULIP effectively improves the performance of multiple recent 3D backbones by simply pre-training them on ShapeNet55 using our framework, achieving state-of-the-art performance in both standard 3D classification and zero-shot 3D classification on ModelNet40 and ScanObjectNN. ULIP also improves the performance of PointMLP by around 3% in 3D classification on ScanObjectNN, and outperforms PointCLIP by 28.8% on top-1 accuracy for zero-shot 3D classification on ModelNet40. Our code and pre-trained models are released at https://github.com/salesforce/ULIP.

Key facts. Cited by 267 papers in the map (data through 14 Oct 2026). Builds on 51 of its 62 references that are in the map. First posted on arXiv 10 Dec 2022.

Builds on

51 of 62 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  6. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  10. 2020Point Transformercited by 1,470
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
38 more
  1. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  2. 2021Grounded Language-Image Pre-trainingcited by 1,156
  3. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  4. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  5. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  6. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 891
  7. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  8. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  9. 2020PCT: Point cloud transformercited by 732
  10. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 657
  11. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  12. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 588
  13. 2019RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Cloudscited by 585
  14. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  15. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  16. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  17. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  18. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 436
  19. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 385
  20. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 379
  21. 2019Relation-Shape Convolutional Neural Network for Point Cloud Analysiscited by 346
  22. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  23. 2018SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filterscited by 261
  24. 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
  25. 2021Group-Free 3D Object Detection via Transformerscited by 215
  26. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 200
  27. 2021Walk in the Cloud: Learning Curves for Point Clouds Shape Analysiscited by 172
  28. 2021Revisiting Point Cloud Shape Classification with a Simple and Effective Baselinecited by 145
  29. 2020MVTN: Multi-View Transformation Network for 3D Shape Recognitioncited by 140
  30. 2021LiDAR R-CNN: An Efficient and Universal 3D Object Detectorcited by 129
  31. 2019DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processingcited by 104
  32. 2019Adaptive Cross-Modal Few-Shot Learningcited by 95
  33. 2022Surface Representation for Point Cloudscited by 84
  34. 2020Learning Geometry-Disentangled Representation for Complementary Understanding of 3D Object Point Cloudcited by 71
  35. 2021Open Vocabulary Object Detection with Pseudo Bounding-Box Labelscited by 71
  36. 2021Learning Inner-Group Relations on Point Cloudscited by 29
  37. 2021Multimodal Semi-Supervised Learning for 3D Objectscited by 24
  38. 2022Let Images Give You More:Point Cloud Cross-Modal Training for Shape Analysiscited by 24

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 40; 2024: 64; 2025: 83; 2026: 80.

267 papers.

Sort:

By area:

  1. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  2. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 356
  3. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  4. 2025SAM 3D: 3Dfy Anything in Imagescited by 247
  5. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
  6. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  7. 2025Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Materialcited by 165
  8. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  9. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  10. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  11. 2025Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Detailscited by 125
  12. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
38 more
  1. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 103
  2. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 95
  3. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  4. 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priorscited by 74
  5. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  6. 2024CAD-Recode: Reverse Engineering CAD Code from Point Cloudscited by 66
  7. 2024Generating CAD Code with Vision-Language Models for 3D Designscited by 62
  8. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  9. 2025AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensorscited by 48
  10. 2025Efficient Part-level 3D Object Generation via Dual Volume Packingcited by 48
  11. 2025LATTICE: Democratize High-Fidelity 3D Generation at Scalecited by 44
  12. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  13. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 39
  14. 2024SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAEcited by 39
  15. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 38
  16. 2023Connecting Multi-modal Contrastive Representationscited by 36
  17. 2023EventCLIP: Adapting CLIP for Event-based Object Recognitioncited by 36
  18. 2023EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understandingcited by 34
  19. 20253D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worldscited by 30
  20. 2023ViT-Lens: Towards Omni-modal Representationscited by 29
  21. 2025Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Priorcited by 29
  22. 2023Point2Vec for Self-Supervised Representation Learning on Point Cloudscited by 28
  23. 2024ContPhy: Continuum Physical Concept Learning and Reasoning from Videoscited by 27
  24. 2024TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understandingcited by 27
  25. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26
  26. 2025Unleashing Vecset Diffusion Model for Fast Shape Generationcited by 26
  27. 2025From One to More: Contextual Part Latents for 3D Generationcited by 26
  28. 2023GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generationcited by 25
  29. 2023Octavius: Mitigating Task Interference in MLLMs via LoRA-MoEcited by 25
  30. 2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24
  31. 2023Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusioncited by 24
  32. 2024M3DM-NR: RGB-3D Noisy-Resistant Industrial Anomaly Detection via Multimodal Denoisingcited by 24
  33. 2024Any2Point: Empowering Any-modality Large Models for Efficient 3D Understandingcited by 23
  34. 2023Hulk: A Universal Knowledge Translator for Human-Centric Taskscited by 22
  35. 2023Self-supervised Learning for Pre-Training 3D Point Clouds: A Surveycited by 21
  36. 2023Extending Multi-modal Contrastive Representationscited by 21
  37. 2024Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban Environmentscited by 21
  38. 2023A Survey of Label-Efficient Deep Learning for 3D Point Cloudscited by 20

50 most cited of 267 All 267 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding
  2. 2024UniPLV: Towards Label-Efficient Open-World 3D Scene Understanding by Regional Visual Language Supervision
  3. 2025Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenization
  4. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  5. 2024ImageNet3D: Towards General-Purpose Object-Level 3D Understanding
  6. 2025Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding
  7. 2025Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models
  8. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding