mlsanityData to 7 Oct 2026

Uni3DL: Unified Model for 3D and Language Understanding

arXiv 2312.03026 ↗First posted 5 Dec 2023cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

In this work, we present Uni3DL, a unified model for 3D and Language understanding. Distinct from existing unified vision-language models in 3D which are limited in task variety and predominantly dependent on projected multi-view images, Uni3DL operates directly on point clouds. This approach significantly expands the range of supported tasks in 3D, encompassing both vision and vision-language tasks in 3D. At the core of Uni3DL, a query transformer is designed to learn task-agnostic semantic and mask outputs by attending to 3D visual features, and a task router is employed to selectively generate task-specific outputs required for diverse tasks. With a unified architecture, our Uni3DL model enjoys seamless task decomposition and substantial parameter sharing across tasks. Uni3DL has been rigorously evaluated across diverse 3D vision-language understanding tasks, including semantic segmentation, object detection, instance segmentation, visual grounding, 3D captioning, and text-3D cross-modal retrieval. It demonstrates performance on par with or surpassing state-of-the-art (SOTA) task-specific models. We hope our benchmark and Uni3DL model will serve as a solid step to ease future research in unified models in the realm of 3D and language understanding. Project page: https://uni3dl.github.io.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 62 of its 85 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

62 of 85 references.

References · 62

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  12. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
38 more
  1. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  2. 2020Object-Centric Learning with Slot Attentioncited by 865
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  5. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  8. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  9. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  10. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  11. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 384
  12. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  13. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  14. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  15. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  16. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  17. 2019Structured3D: A Large Photo-realistic Dataset for Structured 3D Modelingcited by 285
  18. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  19. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  20. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 216
  21. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  22. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  23. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  24. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 197
  25. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  26. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  27. 2020OccuSeg: Occupancy-aware 3D Instance Segmentationcited by 177
  28. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  29. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  30. 2019Learning Object Bounding Boxes for 3D Instance Segmentation on Point Cloudscited by 156
  31. 2022A Unified Sequence Interface for Vision Taskscited by 152
  32. 2018GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloudcited by 145
  33. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  34. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  35. 2021Hierarchical Aggregation for 3D Instance Segmentationcited by 126
  36. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  37. 2022Superpoint Transformer for 3D Scene Instance Segmentationcited by 115
  38. 2023Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understandingcited by 111

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 5.

9 papers.

  1. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  2. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  3. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 24
  4. 2025Multimodal Referring Segmentation: A Surveycited by 22
  5. 2024COM3D: Leveraging Cross-View Correspondence and Cross-Modal Mining for 3D Retrievalcited by 6
  6. 2025PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Groundingcited by 3
  7. 2025Zero-Shot 3D Visual Grounding from Vision-Language Modelscited by 2
  8. 2025Uni-Hema: Unified Model for Digital Hematopathologycited by 2
  9. 2025MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Language-Image Models with 3D Understanding
  2. 2026VLM3: Vision Language Models Are Native 3D Learners
  3. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  4. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight
  5. 20233D-LLM: Injecting the 3D World into Large Language Models
  6. 2025Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understanding
  7. 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
  8. 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation