mlsanityData to 7 Oct 2026

Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

arXiv 2401.06209 ↗First posted 11 Jan 2024cs.CVcited by 719

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Is vision good enough for language? Recent advancements in multimodal models primarily stem from the powerful reasoning abilities of large language models (LLMs). However, the visual component typically depends only on the instance-level contrastive language-image pre-training (CLIP). Our research reveals that the visual capabilities in recent multimodal LLMs (MLLMs) still exhibit systematic shortcomings. To understand the roots of these errors, we explore the gap between the visual embedding space of CLIP and vision-only self-supervised learning. We identify ''CLIP-blind pairs'' - images that CLIP perceives as similar despite their clear visual differences. With these pairs, we construct the Multimodal Visual Patterns (MMVP) benchmark. MMVP exposes areas where state-of-the-art systems, including GPT-4V, struggle with straightforward questions across nine basic visual patterns, often providing incorrect answers and hallucinated explanations. We further evaluate various CLIP-based vision-and-language models and found a notable correlation between visual patterns that challenge CLIP models and those problematic for multimodal LLMs. As an initial effort to address these issues, we propose a Mixture of Features (MoF) approach, demonstrating that integrating vision self-supervised learning features with MLLMs can significantly enhance their visual grounding capabilities. Together, our research suggests visual representation learning remains an open challenge, and accurate visual grounding is crucial for future successful multimodal systems.

Key facts. Cited by 719 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 66 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

51 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  3. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Models19 Sep
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  7. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
32 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  13. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  14. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  15. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  16. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  17. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  18. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  19. 2021ImageNet-21K Pretraining for the Massescited by 607
  20. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  21. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  22. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  23. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  24. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  25. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  26. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  27. 2023Data Filtering Networkscited by 247
  28. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  29. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  30. 2023The effectiveness of MAE pre-pretraining for billion-scale pretrainingcited by 68
  31. 2023Mass-Producing Failures of Multimodal Systems with Language Modelscited by 36
  32. 2023ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracycited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 194; 2025: 270; 2026: 255.

719 papers.

Sort:

By area:

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  2. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  4. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  5. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  6. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  7. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  8. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  9. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  10. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  11. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  12. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
38 more
  1. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  2. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  3. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  4. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  5. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 151
  6. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  7. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  8. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  9. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  10. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  11. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 125
  12. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  13. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 101
  14. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
  15. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  16. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  17. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  18. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 93
  19. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  20. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 84
  21. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 82
  22. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  23. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  24. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  25. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 72
  26. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  27. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  28. 2024Visual Hallucinations of Multi-modal Large Language Modelscited by 68
  29. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  30. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 68
  31. 2025Kwai Keye-VL 1.5 Technical Reportcited by 68
  32. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  33. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  34. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 62
  35. 2025Scaling Language-Free Visual Representation Learningcited by 62
  36. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  37. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  38. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 56

50 most cited of 719 All 719 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMs
  2. 2024Enhancing Advanced Visual Reasoning Ability of Large Language Models
  3. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
  4. 2025Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs
  5. 2025VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
  6. 2024Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
  7. 2025LLMs Can Compensate for Deficiencies in Visual Representations
  8. 2024Instruction Tuning-free Visual Token Complement for Multimodal LLMs