mlsanityData to 13 Oct 2026

Needle In A Multimodal Haystack

arXiv 2406.07230 ↗First posted 11 Jun 2024cs.CV cs.AIcited by 57

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

With the rapid advancement of multimodal large language models (MLLMs), their evaluation has become increasingly comprehensive. However, understanding long multimodal content, as a foundational ability for real-world applications, remains underexplored. In this work, we present Needle In A Multimodal Haystack (MM-NIAH), the first benchmark specifically designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents. Our benchmark includes three types of evaluation tasks: multimodal retrieval, counting, and reasoning. In each task, the model is required to answer the questions according to different key information scattered throughout the given multimodal document. Evaluating the leading MLLMs on MM-NIAH, we observe that existing models still have significant room for improvement on these tasks, especially on vision-centric evaluation. We hope this work can provide a platform for further research on long multimodal document comprehension and contribute to the advancement of MLLMs. Code and benchmark are released at https://github.com/OpenGVLab/MM-NIAH.

Key facts. Cited by 57 papers in the map (data through 13 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 106 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

75 of 106 references.

Closest ancestors · 21

Same direction, newest first

  1. 2024MileBench: Benchmarking MLLMs in Long Context29 Apr
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  3. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension25 Apr
  4. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  5. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  6. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  7. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  9. 2023Generative Multimodal Models are In-Context Learners20 Dec
  10. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models28 Nov
  11. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  12. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
9 more
  1. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  7. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  8. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  6. 2023Visual Instruction Tuningcited by 7,827
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,412
  12. 2023Qwen Technical Reportcited by 3,310
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,703
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,896
  8. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,775
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,709
  10. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,595
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,302
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,295
  13. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,116
  14. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,092
  15. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,082
  16. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  17. 2023VideoChat: Chat-Centric Video Understandingcited by 854
  18. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  19. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 778
  20. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 773
  21. 2024Yi: Open Foundation Models by 01.AIcited by 715
  22. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 706
  23. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 634
  24. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  25. 2024InternLM2 Technical Reportcited by 393
  26. 2024What matters when building vision-language models?cited by 323
  27. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 311
  28. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 310
  29. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  30. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  31. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  32. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  33. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  34. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 115
  35. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  36. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  37. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  38. 2023ControlLLM: Augment Language Models with Tools by Searching on Graphscited by 59

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 15; 2025: 28; 2026: 14.

57 papers.

Sort:

By area:

  1. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 446
  2. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 289
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  4. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 169
  5. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 136
  6. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  7. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  8. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  9. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  10. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  11. 2025Generative Frame Sampler for Long Video Understandingcited by 35
  12. 2024Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reductioncited by 28
38 more
  1. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  2. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  3. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 27
  4. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 24
  5. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  6. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  7. 2025Are Large Vision Language Models Good Game Players?cited by 19
  8. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  9. 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documentscited by 17
  10. 2024Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contextscited by 14
  11. 2024PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Samplingcited by 14
  12. 2024VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by Video Spatiotemporal Augmentationcited by 14
  13. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 13
  14. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  15. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Modelscited by 7
  16. 2025Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMscited by 6
  17. 2025Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmarkcited by 6
  18. 2025Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Modelscited by 6
  19. 2025ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolutioncited by 6
  20. 2025HoPE: Hybrid of Position Embedding for Long Context Vision-Language Modelscited by 4
  21. 2025SolidMark: Evaluating Image Memorization in Generative Modelscited by 3
  22. 2025HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding?cited by 3
  23. 2025Beyond Needle(s) in the Embodied Haystack: Environment, Architecture, and Training Considerations for Long Context Reasoningcited by 3
  24. 2025Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformerscited by 3
  25. 2026MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documentscited by 3
  26. 2026Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reusecited by 3
  27. 2025Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysiscited by 2
  28. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  29. 2025SNS-Bench-VL: Benchmarking Multimodal Large Language Models in Social Networking Servicescited by 2
  30. 2025FRIEDA: Benchmarking Multi-Step Cartographic Reasoning in Vision-Language Modelscited by 2
  31. 2026Contextualized Visual Personalization in Vision-Language Modelscited by 2
  32. 2026Personal Visual Memory from Explicit and Implicit Evidencecited by 2
  33. 2025Two Causally Related Needles in a Video Haystackcited by 1
  34. 2025Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?cited by 1
  35. 2026TS-Haystack: A Multi-Task Retrieval Benchmark for Long-Context Time-Series Reasoningcited by 1
  36. 2026Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Contextcited by 1
  37. 2025Detecting Scarce and Sparse Anomalous: Solving Dual Imbalance in Multi-Instance Learningcited by 0
  38. 2025CroSTAta: Cross-State Transition Attention Transformer for Robotic Manipulationcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
  2. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  3. 2024A Survey on Benchmarks of Multimodal Large Language Models
  4. 2025Finding Needles in Images: Can Multimodal LLMs Locate Fine Details?
  5. 2024Efficient Multimodal Large Language Models: A Survey
  6. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  7. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  8. 2024From Text to Pixel: Advancing Long-Context Understanding in MLLMs