mlsanityData to 14 Oct 2026

Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark

arXiv 2507.15882 ↗First posted 18 Jul 2025cs.CV cs.AI cs.CLcited by 6

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

The proliferation of multimodal Large Language Models has significantly advanced the ability to analyze and understand complex data inputs from different modalities. However, the processing of long documents remains under-explored, largely due to a lack of suitable benchmarks. To address this, we introduce Document Haystack, a comprehensive benchmark designed to evaluate the performance of Vision Language Models (VLMs) on long, visually complex documents. Document Haystack features documents ranging from 5 to 200 pages and strategically inserts pure text or multimodal text+image "needles" at various depths within the documents to challenge VLMs' retrieval capabilities. Comprising 400 document variants and a total of 8,250 questions, it is supported by an objective, automated evaluation framework. We detail the construction and characteristics of the Document Haystack dataset, present results from prominent VLMs and discuss potential research avenues in this area.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 37 references that are in the map. First posted on arXiv 18 Jul 2025.

Builds on

30 of 37 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
18 more
  1. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  2. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  3. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
  4. 2023Large Language Models in Finance: A Surveycited by 226
  5. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 183
  6. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 179
  7. 2024Pixtral 12Bcited by 179
  8. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 163
  9. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 135
  10. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  11. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 88
  12. 2024A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Lawcited by 74
  13. 2024Needle In A Multimodal Haystackcited by 58
  14. 2025The Amazon Nova Family of Models: Technical Report and Model Cardcited by 41
  15. 2023Cross-Modal Retrieval: A Systematic Review of Methods and Future Directionscited by 40
  16. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  17. 2024Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoningcited by 17
  18. 2024Zero-resource Speech Translation and Recognition with LLMscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026SIN-Bench: Tracing Native Evidence Chains in Long-Context Multimodal Scientific Interleaved Literaturecited by 3
  2. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  3. 2026ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extractioncited by 1
  4. 2026VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QAcited by 0
  5. 2026Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agentscited by 0
  6. 2026Automatic multimodal UX improvement recommendations from LLM agent user simulationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents
  2. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
  3. 2025DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding
  4. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
  5. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating
  6. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
  7. 2024Building and better understanding vision-language models: insights and future directions
  8. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding