mlsanityData to 14 Oct 2026

Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts

arXiv 2406.16851 ↗First posted 24 Jun 2024cs.CL cs.AI cs.CVcited by 14

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

We present LoCoVQA, a dynamic benchmark generator for evaluating long-context extractive reasoning in vision language models (VLMs). LoCoVQA augments test examples for mathematical reasoning, VQA, and character recognition tasks with increasingly long visual contexts composed of both in-distribution and out-of-distribution distractor images. Across these tasks, a diverse set of VLMs rapidly lose performance as the visual context length grows, often exhibiting a striking logarithmic decay trend. This test assesses how well VLMs can ignore irrelevant information when answering queries -- a task that is quite easy for language models (LMs) in the text domain -- demonstrating that current state-of-the-art VLMs lack this essential capability for many long-context applications.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 55 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

29 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023Mistral 7Bcited by 3,164
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2024Mixtral of Expertscited by 1,823
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  10. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  12. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
15 more
  1. 2023Instruction Tuning with GPT-4cited by 686
  2. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  3. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  4. 2024What matters when building vision-language models?cited by 326
  5. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  6. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  7. 2023Retrieval meets Long Context Large Language Modelscited by 115
  8. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 88
  9. 2024Needle In A Multimodal Haystackcited by 58
  10. 2023A Survey on Long Text Modeling with Transformerscited by 47
  11. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  12. 2022Investigating Efficiently Extending Transformers for Long Input Summarizationcited by 43
  13. 2024Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMscited by 33
  14. 2024Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQAcited by 9
  15. 2024Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Conceptscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 8.

14 papers.

Sort:

By area:

  1. 2025One ruler to measure them all: Benchmarking multilingual long-context language modelscited by 19
  2. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  3. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  4. 2026More Images, More Problems? A Controlled Analysis of VLM Failure Modescited by 9
  5. 2025Are Vision-Language Models Ready for Dietary Assessment? Exploring the Next Frontier in AI-Powered Food Image Recognitioncited by 7
  6. 2025FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challengingcited by 7
  7. 2025LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understandingcited by 7
  8. 2026Tactile Modality Fusion for Vision-Language-Action Modelscited by 4
  9. 2026WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMscited by 3
  10. 2026Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judgecited by 1
  11. 2026The Effects of Visual Priming on Cooperative Behavior in Vision-Language Modelscited by 0
  12. 2026Real Images, Worse Judgments: Evaluating Vision-Language Models on Concreteness and Imagerycited by 0
2 more
  1. 2026ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Modelscited by 0
  2. 2026The Alignment Illusion in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Models
  2. 2026Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context
  3. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
  4. 2026VTC-R1: Vision-Text Compression for Efficient Long-Context Reasoning
  5. 2026Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?
  6. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  7. 2024NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Vision
  8. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling