mlsanityData to 14 Oct 2026

Finding Needles in Images: Can Multimodal LLMs Locate Fine Details?

arXiv 2508.05053 ↗First posted 7 Aug 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

While Multi-modal Large Language Models (MLLMs) have shown impressive capabilities in document understanding tasks, their ability to locate and reason about fine-grained details within complex documents remains understudied. Consider searching a restaurant menu for a specific nutritional detail or identifying a disclaimer in a lengthy newspaper article tasks that demand careful attention to small but significant details within a broader narrative, akin to Finding Needles in Images (NiM). To address this gap, we introduce NiM, a carefully curated benchmark spanning diverse real-world documents including newspapers, menus, and lecture images, specifically designed to evaluate MLLMs' capability in these intricate tasks. Building on this, we further propose Spot-IT, a simple yet effective approach that enhances MLLMs capability through intelligent patch selection and Gaussian attention, motivated from how humans zoom and focus when searching documents. Our extensive experiments reveal both the capabilities and limitations of current MLLMs in handling fine-grained document understanding tasks, while demonstrating the effectiveness of our approach. Spot-IT achieves significant improvements over baseline methods, particularly in scenarios requiring precise detail extraction from complex layouts.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 50 references that are in the map. First posted on arXiv 7 Aug 2025.

Builds on

20 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024Visual Prompting in Multimodal Large Language Models: A Survey5 Sep
  3. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want29 Mar
  4. 2023Fine-Grained Visual Prompting7 Jun

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  8. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  9. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  10. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  11. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 195
  12. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
4 more
  1. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 111
  2. 2017GP-GAN: Towards Realistic High-Resolution Image Blendingcited by 64
  3. 2020A Survey of Deep Learning Approaches for OCR and Document Understandingcited by 32
  4. 2024Read and Think: An Efficient Step-wise Multimodal Language Model for Document Understanding and Reasoningcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Needle In A Multimodal Haystack
  2. 2025Structured Attention Matters to Multimodal LLMs in Document Understanding
  3. 2026COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts
  4. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
  5. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  6. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
  7. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding
  8. 2026Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs