mlsanityData to 7 Oct 2026

Uni3D: Exploring Unified 3D Representation at Scale

arXiv 2310.06773 ↗First posted 10 Oct 2023cs.CV cs.CLcited by 231

Not on the map. Why?

Cited by 36 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Scaling up representations for images or text has been extensively investigated in the past few years and has led to revolutions in learning vision and language. However, scalable representation for 3D objects and scenes is relatively unexplored. In this work, we present Uni3D, a 3D foundation model to explore the unified 3D representation at scale. Uni3D uses a 2D initialized ViT end-to-end pretrained to align the 3D point cloud features with the image-text aligned features. Via the simple architecture and pretext task, Uni3D can leverage abundant 2D pretrained models as initialization and image-text aligned models as the target, unlocking the great potential of 2D models and scaling-up strategies to the 3D world. We efficiently scale up Uni3D to one billion parameters, and set new records on a broad range of 3D tasks, such as zero-shot classification, few-shot classification, open-world understanding and part segmentation. We show that the strong Uni3D representation also enables applications such as 3D painting and retrieval in the wild. We believe that Uni3D provides a new direction for exploring both scaling up and efficiency of the representation in 3D domain.

Key facts. Cited by 231 papers in the map (data through 7 Oct 2026). Builds on 47 of its 55 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

47 of 55 references.

References · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
35 more
  1. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  5. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  6. 2020Point Transformercited by 1,462
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  8. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  9. 2021Grounded Language-Image Pre-trainingcited by 1,147
  10. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  11. 2016Deep Networks with Stochastic Depthcited by 973
  12. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  13. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  14. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  15. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  16. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  17. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  18. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  19. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  20. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 376
  21. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  22. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  23. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  24. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 306
  25. 2021PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformerscited by 284
  26. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  27. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  28. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  29. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  30. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  31. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  32. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  33. 2020Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentationcited by 112
  34. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  35. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 46; 2025: 91; 2026: 88.

231 papers.

Sort:

By area:

  1. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 389
  2. 2025Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generationcited by 342
  3. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  4. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  5. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  6. 2025Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Materialcited by 157
  7. 2025Native and Compact Structured Latents for 3D Generationcited by 157
  8. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  9. 2025Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Detailscited by 120
  10. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  11. 2025Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attentioncited by 103
  12. 2024MeshXL: Neural Coordinate Field for Generative 3D Foundation Modelscited by 100
38 more
  1. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  2. 2025Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assetscited by 85
  3. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  4. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  5. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Modelscited by 60
  6. 2024DiffGS: Functional Gaussian Splatting Diffusioncited by 50
  7. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  8. 2025Efficient Part-level 3D Object Generation via Dual Volume Packingcited by 47
  9. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  10. 2025ManiFlow: A General Robot Manipulation Policy via Consistency Flow Trainingcited by 43
  11. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  12. 2024Binocular-Guided 3D Gaussian Splatting with View Consistency for Sparse View Synthesiscited by 41
  13. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 41
  14. 2025LATTICE: Democratize High-Fidelity 3D Generation at Scalecited by 41
  15. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  16. 2023Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matchingcited by 35
  17. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  18. 2024Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assemblycited by 34
  19. 2025AutoPartGen: Autogressive 3D Part Generation and Discoverycited by 33
  20. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understandingcited by 32
  21. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 31
  22. 2025MetaScenes: Towards Automated Replica Creation for Real-world 3D Scanscited by 31
  23. 2023GeoDream: Disentangling 2D and Geometric Priors for High-Fidelity and Consistent 3D Generationcited by 30
  24. 20253D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worldscited by 29
  25. 2026LaST0: Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Modelcited by 28
  26. 2024TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understandingcited by 26
  27. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  28. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 26
  29. 2025Unleashing Vecset Diffusion Model for Fast Shape Generationcited by 25
  30. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  31. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  32. 2024UDiFF: Generating Conditional Unsigned Distance Fields with Optimal Wavelet Diffusioncited by 23
  33. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  34. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  35. 2025MaterialRefGS: Reflective Gaussian Splatting with Multi-view Consistent Material Inferencecited by 22
  36. 2025InteractVLM: 3D Interaction Reasoning from 2D Foundational Modelscited by 21
  37. 2025UniLat3D: Geometry-Appearance Unified Latents for Single-Stage 3D Generationcited by 20
  38. 2025URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Modelcited by 20

50 most cited of 231 All 231 (JSON)