mlsanityData to 14 Oct 2026

MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly

arXiv 2505.10610 ↗First posted 15 May 2025cs.CV cs.CLcited by 29

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The rapid extension of context windows in large vision-language models has given rise to long-context vision-language models (LCVLMs), which are capable of handling hundreds of images with interleaved text tokens in a single forward pass. In this work, we introduce MMLongBench, the first benchmark covering a diverse set of long-context vision-language tasks, to evaluate LCVLMs effectively and thoroughly. MMLongBench is composed of 13,331 examples spanning five different categories of downstream tasks, such as Visual RAG and Many-Shot ICL. It also provides broad coverage of image types, including various natural and synthetic images. To assess the robustness of the models to different input lengths, all examples are delivered at five standardized input lengths (8K-128K tokens) via a cross-modal tokenization scheme that combines vision patches and text tokens. Through a thorough benchmarking of 46 closed-source and open-source LCVLMs, we provide a comprehensive analysis of the current models' vision-language long-context ability. Our results show that: i) performance on a single task is a weak proxy for overall long-context capability; ii) both closed-source and open-source models face challenges in long-context vision-language tasks, indicating substantial room for future improvement; iii) models with stronger reasoning ability tend to exhibit better long-context performance. By offering wide task coverage, various image types, and rigorous length control, MMLongBench provides the missing foundation for diagnosing and advancing the next generation of LCVLMs.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 105 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

41 of 105 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024NVILA: Efficient Frontier Visual Language Models5 Dec

Other references · 40

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
28 more
  1. 2025Gemma 3 Technical Reportcited by 1,824
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  4. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  5. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  6. 2024Yi: Open Foundation Models by 01.AIcited by 719
  7. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  8. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  9. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  10. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  11. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  12. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  13. 2024Pixtral 12Bcited by 179
  14. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
  15. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  16. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  17. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 108
  18. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 102
  19. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 88
  20. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 72
  21. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 55
  22. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 55
  23. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  24. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densitiescited by 32
  25. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  26. 2024Localizing Factual Inconsistencies in Attributable Text Generationcited by 14
  27. 2024From Text to Pixel: Advancing Long-Context Understanding in MLLMscited by 9
  28. 2024DivScene: Towards Open-Vocabulary Object Navigation with Large Vision Language Models in Diverse Scenescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 22.

29 papers.

Sort:

By area:

  1. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  2. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Modelscited by 14
  3. 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environmentscited by 14
  4. 2025Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agentscited by 11
  5. 2025MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigmcited by 9
  6. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  7. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  8. 2025SIMPACT: Simulation-Enabled Action Planning using Vision-Language Modelscited by 8
  9. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Modelscited by 7
  10. 2025The Cognitive Bandwidth Bottleneck: Shifting Long-Horizon Agent from Planning with Actions to Planning with Schemascited by 5
  11. 2026Tactile Modality Fusion for Vision-Language-Action Modelscited by 4
  12. 2026Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Headscited by 4
17 more
  1. 2025FC-MIR: A Mobile Screen Awareness Framework for Intent-Aware Recommendation based on Frame-Compressed Multimodal Trajectory Reasoningcited by 2
  2. 2026How to Train Your Long-Context Visual Document Modelcited by 2
  3. 2026Hierarchical Evidence-Driven Reasoning for Long Document Understandingcited by 2
  4. 2026Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judgecited by 1
  5. 2026Fix Before Search: Benchmarking Agentic Query Visual Pre-processing in Multimodal Retrieval-augmented Generationcited by 1
  6. 2026MMCL-Bench: Multimodal Context Learning from Visual Rules, Procedures, and Evidencecited by 1
  7. 2026Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Contextcited by 1
  8. 2026Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Modelscited by 1
  9. 2026CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisitioncited by 1
  10. 2026PMMC: Prospective Multimodal Memory Compilation for Long-Term LVLM Agentscited by 1
  11. 2025MMLongCite: A Benchmark for Evaluating Fidelity of Long-Context Vision-Language Modelscited by 0
  12. 2026Internalized Reasoning for Long-Context Visual Document Understandingcited by 0
  13. 2026MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarioscited by 0
  14. 2026SciLens: Multi-modal Scientific Claim Verification with Agentic Entailment and Groundingcited by 0
  15. 2026How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbationscited by 0
  16. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understandingcited by 0
  17. 2026VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  2. 2024Long Context Transfer from Language to Vision
  3. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?
  4. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?
  5. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
  6. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  7. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  8. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images