mlsanityData to 7 Oct 2026

Uni3D-LLM: Unifying Point Cloud Perception, Generation and Editing with Large Language Models

arXiv 2402.03327 ↗First posted 9 Jan 2024cs.CV cs.AI cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce Uni3D-LLM, a unified framework that leverages a Large Language Model (LLM) to integrate tasks of 3D perception, generation, and editing within point cloud scenes. This framework empowers users to effortlessly generate and modify objects at specified locations within a scene, guided by the versatility of natural language descriptions. Uni3D-LLM harnesses the expressive power of natural language to allow for precise command over the generation and editing of 3D objects, thereby significantly enhancing operational flexibility and controllability. By mapping point cloud into the unified representation space, Uni3D-LLM achieves cross-application functionality, enabling the seamless execution of a wide array of tasks, ranging from the accurate instantiation of 3D objects to the diverse requirements of interactive design. Through a comprehensive suite of rigorous experiments, the efficacy of Uni3D-LLM in the comprehension, generation, and editing of point cloud has been validated. Additionally, we have assessed the impact of integrating a point cloud perception module on the generation and editing processes, confirming the substantial potential of our approach for practical applications.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 49 of its 56 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

49 of 56 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2022A ConvNet for the 2020scited by 3,545
37 more
  1. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  6. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  8. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  11. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  13. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  14. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  15. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  16. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  17. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  18. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  19. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  20. 2017Generative Adversarial Networks: An Overviewcited by 588
  21. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  22. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  23. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  24. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  25. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  26. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  27. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  28. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  29. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  30. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  31. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  32. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  33. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  34. 2021Paint by Wordcited by 107
  35. 2021FCAF3D: Fully Convolutional Anchor-Free 3D Object Detectioncited by 97
  36. 2021NPMs: Neural Parametric Models for 3D Deformable Shapescited by 89
  37. 2019Deep Parametric Shape Predictions using Distance Fieldscited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 13; 2026: 5.

24 papers.

Sort:

By area:

  1. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  2. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 63
  3. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 41
  4. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 31
  5. 2024ChatGarment: Garment Estimation, Generation and Editing via Large Language Modelscited by 22
  6. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  7. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  8. 2025Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generationcited by 10
  9. 2025Beyond Flatlands: Unlocking Spatial Intelligence by Decoupling 3D Reasoning from Numerical Regressioncited by 7
  10. 2025Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentationcited by 5
  11. 2025Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Modelscited by 5
  12. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
12 more
  1. 2025AutoHood3D: A Multi-Modal Benchmark for Automotive Hood Design and Fluid-Structure Interactioncited by 3
  2. 2024COMOGen: A Controllable Text-to-3D Multi-object Generation Frameworkcited by 2
  3. 2025Remove360: Benchmarking Residuals After Object Removal in 3D Gaussian Splattingcited by 2
  4. 2025Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Modelcited by 2
  5. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  6. 20263D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decodingcited by 2
  7. 2025Is there anything left? Measuring semantic residuals of objects removed from 3D Gaussian Splattingcited by 1
  8. 2026SGSoft: Learning Fused Semantic-Geometric Features for 3D Shape Correspondence via Template-Guided Soft Signalscited by 1
  9. 2026ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generationcited by 1
  10. 2024Scene Co-pilot: Procedural Text to Video Generation with Human in the Loopcited by 0
  11. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomycited by 0
  12. 2026Pointy - A Lightweight Transformer for Point Cloud Foundation Modelscited by 0