mlsanityData to 7 Oct 2026

MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs

arXiv 2502.17422 ↗First posted 24 Feb 2025cs.CV cs.AI cs.CLcited by 172

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have experienced rapid progress in visual recognition tasks in recent years. Given their potential integration into many critical applications, it is important to understand the limitations of their visual perception. In this work, we study whether MLLMs can perceive small visual details as effectively as large ones when answering questions about images. We observe that their performance is very sensitive to the size of the visual subject of the question, and further show that this effect is in fact causal by conducting an intervention study. Next, we study the attention patterns of MLLMs when answering visual questions, and intriguingly find that they consistently know where to look, even when they provide the wrong answer. Based on these findings, we then propose training-free visual intervention methods that leverage the internal knowledge of any MLLM itself, in the form of attention and gradient maps, to enhance its perception of small visual details. We evaluate our proposed methods on two widely-used MLLMs and seven visual question answering benchmarks and show that they can significantly improve MLLMs' accuracy without requiring any training. Our results elucidate the risk of applying MLLMs to visual recognition tasks concerning small details and indicate that visual intervention using the model's internal state is a promising direction to mitigate this risk.

Key facts. Cited by 172 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 59 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

52 of 59 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  3. 2024Parrot: Multilingual Visual Instruction Tuning4 Jun
  4. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  5. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models5 Mar
  6. 2024Exploring Perceptual Limitation of Multimodal Large Language Models12 Feb
  7. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  8. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents9 Nov
  9. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  9. 2024GPT-4o System Cardcited by 4,467
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
31 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  8. 2021Grounded Language-Image Pre-trainingcited by 1,147
  9. 2023Vision Transformers Need Registerscited by 844
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  11. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  12. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  13. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  14. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  15. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  16. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  17. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  18. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  19. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  20. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  21. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  22. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  23. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  24. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  25. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  26. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  27. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  28. 2024Matryoshka Query Transformer for Large Vision-Language Modelscited by 49
  29. 2023FIRE: Food Image to REcipe generationcited by 18
  30. 2023A Study of Situational Reasoning for Traffic Understandingcited by 13
  31. 2024Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptionscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 73; 2026: 99.

172 papers.

Sort:

By area:

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 71
  3. 2025Vision Language Models are Biasedcited by 63
  4. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 54
  5. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 47
  6. 2025Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learningcited by 46
  7. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 34
  8. 2025Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecturecited by 30
  9. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
  10. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 28
  11. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  12. 2025FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answeringcited by 20
38 more
  1. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  2. 2025A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Modelscited by 18
  3. 2025SATORI-R1: Incentivizing Multimodal Reasoning through Explicit Visual Anchoringcited by 17
  4. 2025HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decouplingcited by 16
  5. 2025Thinking with Visual Abstract: Enhancing Multimodal Reasoning via Visual Abstractioncited by 15
  6. 2025Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generationcited by 15
  7. 2025CROP: Contextual Region-Oriented Visual Token Pruningcited by 14
  8. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 14
  9. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 14
  10. 2025Thinking in 360deg: Humanoid Visual Search in the Wildcited by 14
  11. 2025See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Modelcited by 13
  12. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  13. 2025Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learningcited by 11
  14. 2025Agentic Learner with Grow-and-Refine Multimodal Semantic Memorycited by 11
  15. 2025Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decodingcited by 10
  16. 2025NeSyGeo: A Neuro-Symbolic Framework for Multimodal Geometric Reasoning Data Generationcited by 10
  17. 2025DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoningcited by 10
  18. 2025Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Groundingcited by 9
  19. 2026Thinking Without Images: Internalizing Visual Manipulation with On-Policy Self-Distillationcited by 9
  20. 2025Position: Reasoning After Perception Means Reasoning Without Visioncited by 8
  21. 2025Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoningcited by 8
  22. 2025Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perceptioncited by 8
  23. 2025Multi-speaker Attention Alignment for Multimodal Social Interactioncited by 8
  24. 2025CropVLM: Learning to Zoom for Fine-Grained Vision-Language Perceptioncited by 8
  25. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 7
  26. 2025Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Predictioncited by 7
  27. 2025LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agentscited by 6
  28. 2025CoFFT: Chain of Foresight-Focus Thought for Visual Language Modelscited by 6
  29. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 6
  30. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 6
  31. 2026SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMscited by 6
  32. 2026Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMscited by 6
  33. 2025The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models? A Bias-Controlled Studycited by 5
  34. 2025Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMscited by 5
  35. 2025CARES: Context-Aware Resolution Selector for VLMscited by 5
  36. 2026Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Modelscited by 5
  37. 2025Visually Interpretable Subtask Reasoning for Visual Question Answeringcited by 4
  38. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4

50 most cited of 172 All 172 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMs
  2. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly
  3. 2024A Survey on Benchmarks of Multimodal Large Language Models
  4. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks
  5. 2025Where do Large Vision-Language Models Look at when Answering Questions?
  6. 2024Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation
  7. 2024Efficient Multimodal Large Language Models: A Survey
  8. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models