mlsanityData to 14 Oct 2026

GPT4Point: A Unified Framework for Point-Language Understanding and Generation

arXiv 2312.02980 ↗First posted 5 Dec 2023cs.CVcited by 71

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have excelled in 2D image-text comprehension and image generation, but their understanding of the 3D world is notably deficient, limiting progress in 3D language understanding and generation. To solve this problem, we introduce GPT4Point, an innovative groundbreaking point-language multimodal model designed specifically for unified 3D object understanding and generation within the MLLM framework. GPT4Point as a powerful 3D MLLM seamlessly can execute a variety of point-text reference tasks such as point-cloud captioning and Q&A. Additionally, GPT4Point is equipped with advanced capabilities for controllable 3D generation, it can get high-quality results through a low-quality point-text feature maintaining the geometric shapes and colors. To support the expansive needs of 3D object-text pairs, we develop Pyramid-XL, a point-language dataset annotation engine. It constructs a large-scale database over 1M objects of varied text granularity levels from the Objaverse-XL dataset, essential for training GPT4Point. A comprehensive benchmark has been proposed to evaluate 3D point-language understanding capabilities. In extensive evaluations, GPT4Point has demonstrated superior performance in understanding and generation.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 84 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

66 of 84 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation6 Aug
  2. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  3. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  7. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  8. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  9. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
  10. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  13. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  14. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  15. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  16. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  17. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  18. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  19. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 687
  20. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  21. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  22. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  23. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  24. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  25. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  26. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  27. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  28. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 442
  29. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  30. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  31. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  32. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 326
  33. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
  34. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  35. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 264
  36. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  37. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  38. 2023Scalable 3D Captioning with Pretrained Modelscited by 203

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 31; 2026: 21.

71 papers.

Sort:

By area:

  1. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  2. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  3. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  4. 2024SAMPart3D: Segment Any Part in 3D Objectscited by 97
  5. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 96
  6. 2025DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Drivingcited by 86
  7. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 70
  8. 2024Multi-modal Situated Reasoning in 3D Scenescited by 51
  9. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  10. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  11. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  12. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 38
38 more
  1. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  2. 2024Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Imagescited by 35
  3. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 35
  4. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 32
  5. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 28
  6. 2025PartNeXt: A Next-Generation Dataset for Fine-Grained and Hierarchical 3D Part Understandingcited by 26
  7. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 25
  8. 2025URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Modelcited by 21
  9. 2024Space3D-Bench: Spatial 3D Question Answering Benchmarkcited by 17
  10. 2024X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Modelscited by 15
  11. 2025CountLLM: Towards Generalizable Repetitive Action Counting via Large Language Modelcited by 14
  12. 2025Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactionscited by 13
  13. 20253DGen-Bench: Comprehensive Benchmark Suite for 3D Generative Modelscited by 12
  14. 2025HaploVL: A Single-Transformer Baseline for Multi-Modal Understandingcited by 11
  15. 2024More Text, Less Point: Towards 3D Data-Efficient Point-Language Understandingcited by 10
  16. 2025PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Modelscited by 8
  17. 20254D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understandingcited by 8
  18. 2025Towards Scalable Spatial Intelligence via 2D-to-3D Data Liftingcited by 8
  19. 2025SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenescited by 8
  20. 2025Part-X-MLLM: Part-aware 3D Multimodal Large Language Modelcited by 7
  21. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 7
  22. 2024LLaNA: Large Language and NeRF Assistantcited by 6
  23. 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understandingcited by 6
  24. 2025Empowering Large Language Models with 3D Situation Awarenesscited by 6
  25. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  26. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 5
  27. 2025Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projectorcited by 5
  28. 20264DPC2hat: Towards Dynamic Point Cloud Understanding with Failure-Aware Bootstrappingcited by 5
  29. 2024Text3DAug -- Prompted Instance Augmentation for LiDAR Perceptioncited by 4
  30. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
  31. 2024Slice-100K: A Multimodal Dataset for Extrusion-based 3D Printingcited by 3
  32. 2025Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Modelscited by 3
  33. 2025Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Modelcited by 2
  34. 2025SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoningcited by 2
  35. 2026PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Modelscited by 2
  36. 2025NeuroVoxel-LM: Language-Aligned 3D Perception via Dynamic Voxelization and Meta-Embeddingcited by 1
  37. 20263D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understandingcited by 1
  38. 2026HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Modelscited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  2. 2026PointLLM-R: Enhancing 3D Point Cloud Reasoning via Chain-of-Thought
  3. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors
  4. 2023Joint Representation Learning for Text and 3D Point Cloud
  5. 20253DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4o
  6. 2026Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations
  7. 2025The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models? A Bias-Controlled Study
  8. 20243DBench: A Scalable 3D Benchmark and Instruction-Tuning Dataset