mlsanityData to 7 Oct 2026

OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding

arXiv 2305.10764 ↗First posted 18 May 2023cs.CVcited by 183

Not on the map. Why?

Cited by 24 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce OpenShape, a method for learning multi-modal joint representations of text, image, and point clouds. We adopt the commonly used multi-modal contrastive learning framework for representation alignment, but with a specific focus on scaling up 3D representations to enable open-world 3D shape understanding. To achieve this, we scale up training data by ensembling multiple 3D datasets and propose several strategies to automatically filter and enrich noisy text descriptions. We also explore and compare strategies for scaling 3D backbone networks and introduce a novel hard negative mining module for more efficient training. We evaluate OpenShape on zero-shot 3D classification benchmarks and demonstrate its superior capabilities for open-world recognition. Specifically, OpenShape achieves a zero-shot accuracy of 46.8% on the 1,156-category Objaverse-LVIS benchmark, compared to less than 10% for existing methods. OpenShape also achieves an accuracy of 85.3% on ModelNet40, outperforming previous zero-shot baseline methods by 20% and performing on par with some fully-supervised methods. Furthermore, we show that our learned embeddings encode a wide range of visual and semantic concepts (e.g., subcategories, color, shape, style) and facilitate fine-grained text-3D and image-3D interactions. Due to their alignment with CLIP embeddings, our learned shape representations can also be integrated with off-the-shelf CLIP-based models for various applications, such as point cloud captioning and point cloud-conditioned image generation.

Key facts. Cited by 183 papers in the map (data through 7 Oct 2026). Builds on 68 of its 85 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

68 of 85 references.

References · 68

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  12. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
38 more
  1. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  2. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  3. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  4. 2021Grounded Language-Image Pre-trainingcited by 1,147
  5. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  6. 2020Contrastive Learning with Hard Negative Samplescited by 619
  7. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  8. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  9. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  10. 2023LERF: Language Embedded Radiance Fieldscited by 538
  11. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  12. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  13. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  14. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  15. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  16. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  17. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  18. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 376
  19. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  20. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  21. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 306
  22. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  23. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  24. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  25. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  26. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  27. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  28. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  29. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  30. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  31. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  32. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Modelscited by 151
  33. 2018PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptorscited by 140
  34. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 117
  35. 2020Self-Supervised Learning for Domain Adaptation on Point-Cloudscited by 99
  36. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  37. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 93
  38. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 47; 2025: 53; 2026: 59.

183 papers.

Sort:

By area:

  1. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  2. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  3. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  4. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  5. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  6. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  7. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  8. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  9. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  10. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  11. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  12. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
38 more
  1. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  2. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  3. 2024InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Priorcited by 110
  4. 2025Sonata: Self-Supervised Learning of Reliable Point Representationscited by 108
  5. 2025Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attentioncited by 103
  6. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 100
  7. 2024I-Design: Personalized LLM Interior Designercited by 87
  8. 2025Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assetscited by 85
  9. 2023UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognitioncited by 64
  10. 2024CAD-Recode: Reverse Engineering CAD Code from Point Cloudscited by 63
  11. 2024EvaGaussians: Event Stream Assisted Gaussian Splatting from Blurry Imagescited by 56
  12. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agentcited by 53
  13. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  14. 2024Point-SAM: Promptable 3D Segmentation Model for Point Cloudscited by 49
  15. 2025Any6D: Model-free 6D Pose Estimation of Novel Objectscited by 47
  16. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  17. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  18. 2024SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Viewscited by 35
  19. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  20. 2023ViT-Lens: Towards Omni-modal Representationscited by 29
  21. 2024TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understandingcited by 26
  22. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26
  23. 2025PICO: Reconstructing 3D People In Contact with Objectscited by 22
  24. 2023Extending Multi-modal Contrastive Representationscited by 21
  25. 2025InteractVLM: 3D Interaction Reasoning from 2D Foundational Modelscited by 21
  26. 2024Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Priorcited by 20
  27. 2024FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusioncited by 19
  28. 2025DexVLG: Dexterous Vision-Language-Grasp Model at Scalecited by 19
  29. 2024SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Mannerscited by 18
  30. 2023Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-trainingcited by 17
  31. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understandingcited by 16
  32. 20233DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4Vcited by 15
  33. 2024UniM-OV3D: Uni-Modality Open-Vocabulary 3D Scene Understanding with Fine-Grained Feature Representationcited by 15
  34. 2024Positional Prompt Tuning for Efficient 3D Representation Learningcited by 15
  35. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  36. 2024EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editingcited by 14
  37. 2024A Survey On Text-to-3D Contents Generation In The Wildcited by 13
  38. 2024OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Drivingcited by 13

50 most cited of 183 All 183 (JSON)