mlsanityData to 14 Oct 2026

From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models

arXiv 2310.08825 ↗First posted 13 Oct 2023cs.CVcited by 98

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multi-modal Large Language Models (MLLMs) have made significant strides in expanding the capabilities of Large Language Models (LLMs) through the incorporation of visual perception interfaces. Despite the emergence of exciting applications and the availability of diverse instruction tuning data, existing approaches often rely on CLIP or its variants as the visual branch, and merely extract features from the deep layers. However, these methods lack a comprehensive analysis of the visual encoders in MLLMs. In this paper, we conduct an extensive investigation into the effectiveness of different vision encoders within MLLMs. Our findings reveal that the shallow layer features of CLIP offer particular advantages for fine-grained tasks such as grounding and region understanding. Surprisingly, the vision-only model DINO, which is not pretrained with text-image alignment, demonstrates promising performance as a visual branch within MLLMs. By simply equipping it with an MLP layer for alignment, DINO surpasses CLIP in fine-grained related perception tasks. Building upon these observations, we propose a simple yet effective feature merging strategy, named COMM, that integrates CLIP and DINO with Multi-level features Merging, to enhance the visual capabilities of MLLMs. We evaluate COMM through comprehensive experiments on a wide range of benchmarks, including image captioning, visual question answering, visual grounding, and object hallucination. Experimental results demonstrate the superior performance of COMM compared to existing methods, showcasing its enhanced visual capabilities within MLLMs.

Key facts. Cited by 98 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 52 references that are in the map. First posted on arXiv 13 Oct 2023.

Builds on

42 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
27 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  4. 2015VQA: Visual Question Answeringcited by 2,506
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  12. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  13. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  14. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  15. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  16. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  17. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  18. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  19. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  20. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  21. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  22. 2022DeiT III: Revenge of the ViTcited by 419
  23. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  24. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  25. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  26. 2016A Joint Speaker-Listener-Reinforcer Model for Referring Expressionscited by 74
  27. 2020Point and Ask: Incorporating Pointing into Visual Question Answeringcited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 24; 2025: 39; 2026: 34.

98 papers.

Sort:

By area:

  1. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  2. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  3. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 119
  4. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  5. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 68
  6. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 53
  7. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  8. 2024Dense Connector for MLLMscited by 49
  9. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 46
  10. 2024Diffusion Feedback Helps CLIP See Bettercited by 38
  11. 2025Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Modelscited by 37
  12. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
38 more
  1. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approachcited by 35
  2. 2024CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathologycited by 34
  3. 2024Libra: Leveraging Temporal Images for Biomedical Radiology Analysiscited by 29
  4. 2025Interpretable and Testable Vision Features via Sparse Autoencoderscited by 25
  5. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understandingcited by 23
  6. 2025Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practicescited by 22
  7. 2025Toward Generalizable Forgery Detection and Reasoningcited by 20
  8. 2025Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Modelscited by 15
  9. 2026SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoningcited by 15
  10. 2024USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentationcited by 14
  11. 2026When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigationcited by 14
  12. 2026Learning Invariant Visual Representations for Planning with Joint-Embedding Predictive World Modelscited by 13
  13. 2024Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Modelscited by 11
  14. 2024RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representationscited by 11
  15. 2025Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Modelscited by 11
  16. 2025HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policiescited by 10
  17. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervisioncited by 9
  18. 2025Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attackscited by 8
  19. 2026FASA: Frequency-aware Sparse Attentioncited by 8
  20. 2025Connector-S: A Survey of Connectors in Multi-modal Large Language Modelscited by 7
  21. 2025Visual Persona: Foundation Model for Full-Body Human Customizationcited by 7
  22. 2025MM-DINOv2: Adapting Foundation Models for Multi-Modal Medical Image Analysiscited by 7
  23. 2026Attention Smoothing Is All You Need For Unlearningcited by 7
  24. 2026PASA: A Principled Embedding-Space Watermarking Approach for LLM-Generated Text under Semantic-Invariant Attackscited by 7
  25. 2024OmniFusion Technical Reportcited by 6
  26. 2024HybridBooth: Hybrid Prompt Inversion for Efficient Subject-Driven Generationcited by 6
  27. 2025Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimationcited by 6
  28. 2025MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoningcited by 6
  29. 2024KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehensioncited by 5
  30. 2025Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoderscited by 5
  31. 2025Data Scaling Laws for Radiology Foundation Modelscited by 5
  32. 2025FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Managementcited by 5
  33. 2025VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMscited by 5
  34. 2024ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editingcited by 4
  35. 2025What's in a Latent? Leveraging Diffusion Latent Space for Domain Generalizationcited by 3
  36. 2025Visual Language Models show widespread visual deficits on neuropsychological testscited by 3
  37. 2025Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractorcited by 3
  38. 2025METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Modelscited by 3

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Models
  2. 2026Language-Conditioned Visual Grounding with CLIP Multilingual
  3. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder
  4. 2025QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retraining
  5. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
  6. 2025LLMs Can Compensate for Deficiencies in Visual Representations
  7. 2025Data or Language Supervision: What Makes CLIP Better than DINO?
  8. 2025Multimodal Language Models See Better When They Look Shallower