mlsanityData to 7 Oct 2026

BLINK: Multimodal Large Language Models Can See but Not Perceive

arXiv 2404.12390 ↗First posted 18 Apr 2024cs.CV cs.AI cs.CLcited by 529

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce Blink, a new benchmark for multimodal language models (LLMs) that focuses on core visual perception abilities not found in other evaluations. Most of the Blink tasks can be solved by humans "within a blink" (e.g., relative depth estimation, visual correspondence, forensics detection, and multi-view reasoning). However, we find these perception-demanding tasks cast significant challenges for current multimodal LLMs because they resist mediation through natural language. Blink reformats 14 classic computer vision tasks into 3,807 multiple-choice questions, paired with single or multiple images and visual prompting. While humans get 95.70% accuracy on average, Blink is surprisingly challenging for existing multimodal LLMs: even the best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only 13.17% and 7.63% higher than random guessing, indicating that such perception abilities have not "emerged" yet in recent multimodal LLMs. Our analysis also highlights that specialist CV models could solve these problems much better, suggesting potential pathways for future improvements. We believe Blink will stimulate the community to help multimodal LLMs catch up with human-level visual perception.

Key facts. Cited by 529 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 86 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

60 of 86 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  2. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models28 Nov
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  9. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  12. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  13. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  14. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  15. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  16. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  17. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  18. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  19. 2023Emergent Correspondence from Image Diffusioncited by 437
  20. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  21. 2023DIRE for Diffusion-Generated Image Detectioncited by 386
  22. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  23. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  24. 2022Visual Spatial Reasoningcited by 288
  25. 2017HPatches: A benchmark and evaluation of handcrafted and learned local descriptorscited by 261
  26. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  27. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  28. 2016Single-Image Depth Perception in the Wildcited by 179
  29. 2018TallyQA: Answering Complex Counting Questionscited by 152
  30. 2019SPair-71k: A Large-scale Benchmark for Semantic Correspondencecited by 140
  31. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  32. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66
  33. 2022Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Datasetcited by 59
  34. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  35. 2023Intrinsic Image Decomposition via Ordinal Shadingcited by 53
  36. 2022There is a Time and Place for Reasoning Beyond the Imagecited by 22
  37. 2021The Functional Correspondence Problemcited by 20
  38. 2023Generate then Select: Open-ended Visual Question Answering Guided by World Knowledgecited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 83; 2025: 188; 2026: 258.

529 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2025Qwen3-VL Technical Reportcited by 2,534
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  4. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  5. 2025Gemma 3 Technical Reportcited by 1,686
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  8. 2024Depth Anything V2cited by 1,385
  9. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  10. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  11. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  12. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
38 more
  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  2. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  3. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  4. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  5. 2025Kimi-VL Technical Reportcited by 381
  6. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  7. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  8. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  9. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  10. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  11. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  12. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  13. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  14. 2025Thyme: Think Beyond Imagescited by 160
  15. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 151
  16. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
  17. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  18. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  19. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  20. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 120
  21. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  22. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 111
  23. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  24. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  25. 2025MiMo-VL Technical Reportcited by 106
  26. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  27. 2025Ovis2.5 Technical Reportcited by 103
  28. 2025Latent Visual Reasoningcited by 99
  29. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  30. 2025Visual Spatial Tuningcited by 85
  31. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  32. 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 81
  33. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  34. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  35. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  36. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  37. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  38. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 68

50 most cited of 529 All 529 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies
  2. 2025Blink: Dynamic Visual Token Resolution for Enhanced Multimodal Understanding
  3. 2024Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
  4. 2026Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
  6. 2025Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs
  7. 2023Lost in Translation: When GPT-4V(ision) Can't See Eye to Eye with Text. A Vision-Language-Consistency Analysis of VLLMs and Beyond
  8. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions