# MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors

arXiv: https://arxiv.org/abs/2405.01413 · HTML: https://mlsanity.com/paper/2405.01413/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 33 of its 60 references that are in the map.
- First posted on arXiv 2 May 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition](https://mlsanity.com/paper/2309.15112/) (2023)
- [LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding](https://mlsanity.com/paper/2306.17107/) (2023)
- [From Sparse to Soft Mixtures of Experts](https://mlsanity.com/paper/2308.00951/) (2023)
- [LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning](https://mlsanity.com/paper/2311.18651/) (2023)
- [ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding](https://mlsanity.com/paper/2305.08275/) (2023)
- [Honeybee: Locality-enhanced Projector for Multimodal LLM](https://mlsanity.com/paper/2312.06742/) (2023)
- [Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following](https://mlsanity.com/paper/2309.00615/) (2023)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning](https://mlsanity.com/paper/2309.05444/) (2023)
- [Higher Layers Need More LoRA Experts](https://mlsanity.com/paper/2402.08562/) (2024)
- [Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles](https://mlsanity.com/paper/2309.10228/) (2023)
- [Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference](https://mlsanity.com/paper/2110.03742/) (2021)

## Built on by (most cited first, 30 of 49)

- [3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding](https://mlsanity.com/paper/2506.01946/) (2025)
- [LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences](https://mlsanity.com/paper/2412.01292/) (2024)
- [3D Aware Region Prompted Vision Language Model](https://mlsanity.com/paper/2509.13317/) (2025)
- [Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model](https://mlsanity.com/paper/2405.17427/) (2024)
- [A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directions](https://mlsanity.com/paper/2406.05785/) (2024)
- [Exploring the Potential of Encoder-free Architectures in 3D LMMs](https://mlsanity.com/paper/2502.09620/) (2025)
- [Duoduo CLIP: Efficient 3D Understanding with Multi-View Images](https://mlsanity.com/paper/2406.11579/) (2024)
- [Multi-GraspLLM: A Multimodal LLM for Multi-Hand Semantic Guided Grasp Generation](https://mlsanity.com/paper/2412.08468/) (2024)
- [Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Vision](https://mlsanity.com/paper/2506.03605/) (2025)
- [More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding](https://mlsanity.com/paper/2408.15966/) (2024)
- [Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2504.19500/) (2025)
- [Kestrel: 3D Multimodal LLM for Part-Aware Grounded Description](https://mlsanity.com/paper/2405.18937/) (2024)
- [PerLA: Perceptive 3D Language Assistant](https://mlsanity.com/paper/2411.19774/) (2024)
- [Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs](https://mlsanity.com/paper/2506.05318/) (2025)
- [Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting](https://mlsanity.com/paper/2507.18678/) (2025)
- [MICAS: Multi-grained In-Context Adaptive Sampling for 3D Point Cloud Processing](https://mlsanity.com/paper/2411.16773/) (2024)
- [4DPC2hat: Towards Dynamic Point Cloud Understanding with Failure-Aware Bootstrapping](https://mlsanity.com/paper/2602.03890/) (2026)
- [Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts](https://mlsanity.com/paper/2505.21079/) (2025)
- [Aligning Text, Images, and 3D Structure Token-by-Token](https://mlsanity.com/paper/2506.08002/) (2025)
- [Foundation Models for Autonomous Robots in Unstructured Environments](https://mlsanity.com/paper/2407.14296/) (2024)
- [GeoProg3D: Compositional Visual Reasoning for City-Scale 3D Language Fields](https://mlsanity.com/paper/2506.23352/) (2025)
- [AutoHood3D: A Multi-Modal Benchmark for Automotive Hood Design and Fluid-Structure Interaction](https://mlsanity.com/paper/2511.05596/) (2025)
- [ArtiWorld: LLM-Driven Articulation of 3D Objects in Scenes](https://mlsanity.com/paper/2511.12977/) (2025)
- [BrepCoder: A Unified Multimodal Large Language Model for Multi-task B-rep Reasoning](https://mlsanity.com/paper/2602.22284/) (2026)
- [EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformers](https://mlsanity.com/paper/2605.16745/) (2026)
- [Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding](https://mlsanity.com/paper/2507.09334/) (2025)
- [Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Model](https://mlsanity.com/paper/2509.07825/) (2025)
- [PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models](https://mlsanity.com/paper/2603.00412/) (2026)
- [Point Cloud as a Foreign Language for Multi-modal Large Language Model](https://mlsanity.com/paper/2603.09173/) (2026)
- [Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding](https://mlsanity.com/paper/2512.12822/) (2025)

## Abstract (arXiv)

Large 2D vision-language models (2D-LLMs) have gained significant attention by bridging Large Language Models (LLMs) with images using a simple projector. Inspired by their success, large 3D point cloud-language models (3D-LLMs) also integrate point clouds into LLMs. However, directly aligning point clouds with LLM requires expensive training costs, typically in hundreds of GPU-hours on A100, which hinders the development of 3D-LLMs. In this paper, we introduce MiniGPT-3D, an efficient and powerful 3D-LLM that achieves multiple SOTA results while training for only 27 hours on one RTX 3090. Specifically, we propose to align 3D point clouds with LLMs using 2D priors from 2D-LLMs, which can leverage the similarity between 2D and 3D visual information. We introduce a novel four-stage training strategy for modality alignment in a cascaded way, and a mixture of query experts module to adaptively aggregate features with high efficiency. Moreover, we utilize parameter-efficient fine-tuning methods LoRA and Norm fine-tuning, resulting in only 47.8M learnable parameters, which is up to 260x fewer than existing methods. Extensive experiments show that MiniGPT-3D achieves SOTA on 3D object classification and captioning tasks, with significantly cheaper training costs. Notably, MiniGPT-3D gains an 8.12 increase on GPT-4 evaluation score for the challenging object captioning task compared to ShapeLLM-13B, while the latter costs 160 total GPU-hours on 8 A800. We are the first to explore the efficient 3D-LLM, offering new insights to the community. Code and weights are available at https://github.com/TangYuan96/MiniGPT-3D.