mlsanityData to 14 Oct 2026

Projector Is All You Train

arXiv 2608.19726 ↗First posted 20 Aug 2026cs.CL cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The typical training process of a multimodal large language model (MLLM) involves adapting both the language model backbone and the projector between the backbone and a modality-specific encoder. We ask whether fine-tuning the backbone of an MLLM is necessary to adapt it to a new modality. Through experiments on 3D MLLMs, we find that training only the projector is sufficient to achieve strong multimodal performance relative to existing baseline models and our jointly trained MLLMs with the same encoder and backbone. We also show that joint training leads to undesirable drift in existing capabilities of the language model, which projector-only training avoids by definition. Furthermore, projector-only training has approximately twice the training sample throughput of joint training. We validate our findings across different language model backbones via 3D classification and captioning benchmarks as well as standard benchmarks evaluating language, vision, and spatial reasoning capabilities.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 49 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

38 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026BabyVision: Visual Reasoning Beyond Language10 Jan
  2. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Models19 Sep
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
23 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  6. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  7. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  8. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  9. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  10. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
  11. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  12. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 326
  13. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  14. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 202
  15. 2024Are We Done with MMLU?cited by 194
  16. 2025Generalizing Verifiable Instruction Followingcited by 194
  17. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  18. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 145
  19. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  20. 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Roboticscited by 127
  21. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 107
  22. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 49
  23. 2026PointLLM-R: Enhancing 3D Point Cloud Reasoning via Chain-of-Thoughtcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Projector Training in Multimodal LLMs
  2. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
  3. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  4. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
  5. 2024The Revolution of Multimodal Large Language Models: A Survey
  6. 2024On Domain-Adaptive Post-Training for Multimodal Large Language Models
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
  8. 2025On Path to Multimodal Generalist: General-Level and General-Bench