mlsanityData to 7 Oct 2026

Exploring the Potential of Encoder-free Architectures in 3D LMMs

arXiv 2502.09620 ↗First posted 13 Feb 2025cs.CV cs.AI cs.CLcited by 16

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Encoder-free architectures have been preliminarily explored in the 2D Large Multimodal Models (LMMs), yet it remains an open question whether they can be effectively applied to 3D understanding scenarios. In this paper, we present the first comprehensive investigation into the potential of encoder-free architectures to alleviate the challenges of encoder-based 3D LMMs. These long-standing challenges include the failure to adapt to varying point cloud resolutions during inference and the point features from the encoder not meeting the semantic needs of Large Language Models (LLMs). We identify key aspects for 3D LMMs to remove the pre-trained encoder and enable the LLM to assume the role of the 3D encoder: 1) We propose the LLM-embedded Semantic Encoding strategy in the pre-training stage, exploring the effects of various point cloud self-supervised losses. And we present the Hybrid Semantic Loss to extract high-level semantics. 2) We introduce the Hierarchical Geometry Aggregation strategy in the instruction tuning stage. This incorporates inductive bias into the LLM layers to focus on the local details of the point clouds. To the end, we present the first Encoder-free 3D LMM, ENEL. Our 7B model rivals the state-of-the-art model, PointLLM-PiSA-13B, achieving 57.91%, 61.0%, and 55.20% on the classification, captioning, and VQA tasks, respectively. Our results show that the encoder-free architecture is highly promising for replacing encoder-based architectures in the field of 3D understanding. The code is released at https://github.com/Ivan-Tang-3D/ENEL

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 58 references that are in the map. First posted on arXiv 13 Feb 2025.

Builds on

44 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors2 May
  2. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interaction27 Feb

Other references · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2023Mistral 7Bcited by 3,065
30 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2020Supervised Contrastive Learningcited by 2,803
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  4. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  5. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  6. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  7. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  8. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  9. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  10. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  11. 2024InternLM2 Technical Reportcited by 391
  12. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  13. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  14. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  15. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  16. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  17. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  18. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  19. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  20. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  21. 2023PointGPT: Auto-regressively Generative Pre-training from Point Cloudscited by 115
  22. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  23. 2024Unveiling Encoder-Free Vision-Language Modelscited by 90
  24. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  25. 2023Point-PEFT: Parameter-Efficient Fine-Tuning for 3D Pre-trained Modelscited by 42
  26. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 38
  27. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 35
  28. 2024Any2Point: Empowering Any-modality Large Models for Efficient 3D Understandingcited by 23
  29. 2024More Text, Less Point: Towards 3D Data-Efficient Point-Language Understandingcited by 9
  30. 2024MM-Mixing: Multi-Modal Mixing Alignment for 3D Understandingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 5.

16 papers.

Sort:

By area:

  1. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 45
  2. 2025Taming Anomalies with Down-Up Sampling Networks: Group Center Preserving Reconstruction for 3D Anomaly Detectioncited by 11
  3. 2025AerialVG: A Challenging Benchmark for Aerial Visual Grounding by Exploring Positional Relationscited by 10
  4. 2025PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Modelscited by 8
  5. 2025Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMscited by 8
  6. 2025StyleMotif: Multi-Modal Motion Stylization using Style-Content Cross Fusioncited by 7
  7. 2025Fence Theorem: Towards Dual-Objective Semantic-Structure Isolation in Preprocessing Phase for 3D Anomaly Detectioncited by 4
  8. 2025A Review of 3D Object Detection with Vision-Language Modelscited by 4
  9. 2025Examining the Source of Defects from a Mechanical Perspective for 3D Anomaly Detectioncited by 4
  10. 2025AutoMat: Enabling Automated Crystal Structure Reconstruction from Microscopy via Agentic Tool Usecited by 3
  11. 2025REVISION:Reflective Intent Mining and Online Reasoning Auxiliary for E-commerce Visual Search System Optimizationcited by 1
  12. 2026Efficient Encoder-Free Fourier-based 3D Large Multimodal Modelcited by 1
4 more
  1. 2026Point Cloud as a Foreign Language for Multi-modal Large Language Modelcited by 1
  2. 2026Learning Discriminative Signed Distance Functions from Multi-scale Level-of-detail Features for 3D Anomaly Detectioncited by 0
  3. 2026VGGT-Edit: Feed-forward Native 3D Scene Editing with Residual Field Predictioncited by 0
  4. 2026How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  2. 2025Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding
  3. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning
  4. 20243DBench: A Scalable 3D Benchmark and Instruction-Tuning Dataset
  5. 20263D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models
  6. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding
  7. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLM
  8. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning