mlsanityData to 7 Oct 2026

Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models

arXiv 2406.11230 ↗First posted 17 Jun 2024cs.LG cs.AI cs.CLcited by 53

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have shown significant promise in various applications, leading to broad interest from researchers and practitioners alike. However, a comprehensive evaluation of their long-context capabilities remains underexplored. To address these gaps, we introduce the MultiModal Needle-in-a-haystack (MMNeedle) benchmark, specifically designed to assess the long-context capabilities of MLLMs. Besides multi-image input, we employ image stitching to further increase the input context length, and develop a protocol to automatically generate labels for sub-image level retrieval. Essentially, MMNeedle evaluates MLLMs by stress-testing their capability to locate a target sub-image (needle) within a set of images (haystack) based on textual instructions and descriptions of image contents. This setup necessitates an advanced understanding of extensive visual contexts and effective information retrieval within long-context image inputs. With this benchmark, we evaluate state-of-the-art MLLMs, encompassing both API-based and open-source models. The findings reveal that GPT-4o consistently surpasses other models in long-context scenarios, but suffers from hallucination problems in negative samples, i.e., when needles are not in the haystacks. Our comprehensive long-context evaluation of MLLMs also sheds lights on the considerable performance gap between API-based and open-source models. All the code, data, and instructions required to reproduce the main results are available at https://github.com/Wang-ML-Lab/multimodal-needle-in-a-haystack.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 37 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

24 of 37 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models3 May
  2. 2024MileBench: Benchmarking MLLMs in Long Context29 Apr
  3. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  4. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  5. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  9. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  10. 2024What matters when building vision-language models?cited by 321
  11. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  12. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
4 more
  1. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 79
  2. 2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaborationcited by 38
  3. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  4. 2019Needles in Haystacks: On Classifying Tiny Objects in Large Imagescited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 26; 2026: 17.

53 papers.

Sort:

By area:

  1. 2024Long Context Transfer from Language to Visioncited by 502
  2. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  3. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  4. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  5. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  6. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  7. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMscited by 47
  8. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 30
  9. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  10. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  11. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  12. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 24
38 more
  1. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  2. 2025SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Modelscited by 18
  3. 2025Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understandingcited by 18
  4. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  5. 2024VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by Video Spatiotemporal Augmentationcited by 14
  6. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  7. 2025Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMscited by 9
  8. 2025LingoLoop Attack: Trapping MLLMs via Linguistic Context and State Entrapment into Endless Loopscited by 9
  9. 2024MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmarkcited by 8
  10. 2024Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-Oasiscited by 8
  11. 2025Guided Code Generation with LLMs: A Multi-Agent Framework for Complex Code Taskscited by 8
  12. 2025NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tablescited by 8
  13. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  14. 2025Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Modelscited by 6
  15. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  16. 2024The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMscited by 4
  17. 2025HoPE: Hybrid of Position Embedding for Long Context Vision-Language Modelscited by 4
  18. 2025Movie Facts and Fibs (MF2): A Benchmark for Long Movie Understandingcited by 4
  19. 2025Beyond Needle(s) in the Embodied Haystack: Environment, Architecture, and Training Considerations for Long Context Reasoningcited by 3
  20. 2025Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universecited by 3
  21. 2026MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documentscited by 3
  22. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Modelscited by 3
  23. 2025From Evaluation to Defense: Advancing Safety in Video Large Language Modelscited by 2
  24. 2025Resource Consumption Red-Teaming for Large Vision-Language Modelscited by 2
  25. 2025Towards Fine-Grained Recognition with Large Visual Language Models: Benchmark and Optimization Strategiescited by 2
  26. 2026AI and My Values: User Perceptions of LLMs' Ability to Extract, Embody, and Explain Human Values from Casual Conversationscited by 2
  27. 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMscited by 2
  28. 2026Personal Visual Memory from Explicit and Implicit Evidencecited by 2
  29. 2026Contextualized Visual Personalization in Vision-Language Modelscited by 1
  30. 2026TS-Haystack: A Multi-Task Retrieval Benchmark for Long-Context Time-Series Reasoningcited by 1
  31. 2026Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Contextcited by 1
  32. 2025Eka-Eval: An Evaluation Framework for Low-Resource Multilingual Large Language Modelscited by 0
  33. 2025Struc-EMB: The Potential of Structure-Aware Encoding in Language Embeddingscited by 0
  34. 2025Retrieval Quality at Context Limitcited by 0
  35. 2026Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Modelscited by 0
  36. 2026BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answeringcited by 0
  37. 2026Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibrationcited by 0
  38. 2026Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Gamescited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Needle In A Multimodal Haystack
  2. 2026Indexing Multimodal Language Models for Large-scale Image Retrieval
  3. 2024From Text to Pixel: Advancing Long-Context Understanding in MLLMs
  4. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  5. 2025Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark
  6. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  7. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images
  8. 2025ImageScope: Unifying Language-Guided Image Retrieval via Large Multimodal Model Collective Reasoning