mlsanityData to 14 Oct 2026

Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark

arXiv 2407.13766 ↗First posted 18 Jul 2024cs.CVcited by 55

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

Large Multimodal Models (LMMs) have made significant strides in visual question-answering for single images. Recent advancements like long-context LMMs have allowed them to ingest larger, or even multiple, images. However, the ability to process a large number of visual tokens does not guarantee effective retrieval and reasoning for multi-image question answering (MIQA), especially in real-world applications like photo album searches or satellite imagery analysis. In this work, we first assess the limitations of current benchmarks for long-context LMMs. We address these limitations by introducing a new vision-centric, long-context benchmark, "Visual Haystacks (VHs)". We comprehensively evaluate both open-source and proprietary models on VHs, and demonstrate that these models struggle when reasoning across potentially unrelated images, perform poorly on cross-image reasoning, as well as exhibit biases based on the placement of key information within the context window. Towards a solution, we introduce MIRAGE (Multi-Image Retrieval Augmented Generation), an open-source, lightweight visual-RAG framework that processes up to 10k images on a single 40G A100 GPU -- far surpassing the 1k-image limit of contemporary models. MIRAGE demonstrates up to 13% performance improvement over existing open-source LMMs on VHs, sets a new state-of-the-art on the RetVQA multi-image QA benchmark, and achieves competitive performance on single-image QA with state-of-the-art LMMs. Our dataset, model, and code are available at: https://visual-haystacks.github.io.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 70 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

55 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Retrieval-Augmented Multimodal Language Modeling22 Nov

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
38 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  12. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  13. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  14. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  15. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  16. 2022MAGVIT: Masked Generative Video Transformercited by 346
  17. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  18. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
  19. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  20. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220
  21. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  22. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  23. 2024Are We Done with MMLU?cited by 194
  24. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  25. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 179
  26. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 179
  27. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 151
  28. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  29. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  30. 2023Retrieval meets Long Context Large Language Modelscited by 115
  31. 2021WebQA: Multihop and Multimodal QAcited by 109
  32. 2024Needle In A Multimodal Haystackcited by 58
  33. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  34. 2021Document Collection Visual Question Answeringcited by 45
  35. 2022Gender and Racial Bias in Visual Question Answering Datasetscited by 39
  36. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  37. 2019Exposing and Correcting the Gender Bias in Image Captioning Datasets and Modelscited by 36
  38. 2020Visual Question Answering on Image Setscited by 26

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 23; 2026: 30.

55 papers.

Sort:

By area:

  1. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 183
  2. 2025Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?cited by 76
  3. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 65
  4. 2025Search Arena: Analyzing Search-Augmented LLMscited by 34
  5. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 33
  6. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  7. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 27
  8. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Framescited by 27
  9. 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documentscited by 17
  10. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  11. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 12
  12. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agentscited by 11
38 more
  1. 2025VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generationcited by 10
  2. 2025ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpuscited by 10
  3. 2026More Images, More Problems? A Controlled Analysis of VLM Failure Modescited by 9
  4. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 8
  5. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  6. 2025REN: Fast and Efficient Region Encodings from Patch-Based Image Encoderscited by 5
  7. 2025From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generationcited by 4
  8. 2025Thinking with Images via Self-Calling Agentcited by 4
  9. 2025HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding?cited by 3
  10. 2026The Side Effects of Being Smart: Safety Risks in MLLMs' Multi-Image Reasoningcited by 3
  11. 2024GIRAFFE: Design Choices for Extending the Context Length of Visual Language Modelscited by 2
  12. 2025Mind the (Language) Gap: Towards Probing Numerical and Cross-Lingual Limits of LVLMscited by 2
  13. 2025MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generationcited by 2
  14. 2025Multimodal LLMs for Historical Dataset Construction from Archival Image Scans: German Patents (1877-1918)cited by 2
  15. 2025Towards Long-window Anchoring in Vision-Language Model Distillationcited by 2
  16. 2026T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalabilitycited by 2
  17. 2026Personal Visual Memory from Explicit and Implicit Evidencecited by 2
  18. 2026Online Video Agent Harness for Long Video Understandingcited by 2
  19. 2026Why Do Vision Language Models Struggle To Recognize Human Emotions?cited by 1
  20. 2026Utility-Oriented Visual Evidence Selection for Multimodal Retrieval-Augmented Generationcited by 1
  21. 2026Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Contextcited by 1
  22. 2026Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editingcited by 1
  23. 2025Neural Catalog: Scaling Species Recognition with Catalog of Life-Augmented Generationcited by 0
  24. 2025What's in Common? Multimodal Models Hallucinate When Reasoning Across Scenescited by 0
  25. 2026Decoding the Pulse of Reasoning VLMs in Multi-Image Understanding Taskscited by 0
  26. 2026How to Utilize Complementary Vision-Text Information for 2D Structure Understandingcited by 0
  27. 2026Grid2Matrix: Revealing Digital Agnosia in Vision-Language Modelscited by 0
  28. 2026Personal AI Agent for Camera Roll VQAcited by 0
  29. 2026Task-Aware Structured Memory for Dynamic Multi-modal In-Context Learningcited by 0
  30. 2026Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneitycited by 0
  31. 2026ReToken: One Token to Improve Vision-Language Models for Visual Retrievalcited by 0
  32. 2026StateTrace: An Object-Centric Framework for Hidden-State Spatiotemporal Reasoning in Long Videoscited by 0
  33. 2026Fine-Grained Multi Image Object Hallucination Benchmarkcited by 0
  34. 2026Learning to Zoom Efficiently with a Contrastive Curriculumcited by 0
  35. 2026CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generationcited by 0
  36. 2026Collaborative Memory for Multi-Agent VLM Systemscited by 0
  37. 2026C3M: Cross-Session Multimodal Memory Maintenance for Long-Horizon Taskscited by 0
  38. 2026Can't Find Waldo: Evaluating VLMs' Sensitivity to Image Resolution and Detail Levelcited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AeroRAG: Structured Multimodal Retrieval-Augmented LLM for Fine-Grained Aerial Visual Reasoning
  2. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  3. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
  4. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models
  5. 2025Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
  6. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queries
  7. 2026KIRA: Knowledge-Intensive Image Retrieval and Reasoning Architecture for Specialized Visual Domains
  8. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models