# MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly

arXiv: https://arxiv.org/abs/2505.10610 · HTML: https://mlsanity.com/paper/2505.10610/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 105 references that are in the map.
- First posted on arXiv 15 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 41)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [LongVILA: Scaling Long-Context Visual Language Models for Long Videos](https://mlsanity.com/paper/2408.10188/) (2024)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [NVILA: Efficient Frontier Visual Language Models](https://mlsanity.com/paper/2412.04468/) (2024)
- [Pixtral 12B](https://mlsanity.com/paper/2410.07073/) (2024)
- [Ovis: Structural Embedding Alignment for Multimodal Large Language Model](https://mlsanity.com/paper/2405.20797/) (2024)
- [How to Train Long-Context Language Models (Effectively)](https://mlsanity.com/paper/2410.02660/) (2024)
- [HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly](https://mlsanity.com/paper/2410.02694/) (2024)
- [LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs](https://mlsanity.com/paper/2406.15319/) (2024)

## Built on by (most cited first, 29 of 29)

- [Seed1.8 Model Card: Towards Generalized Real-World Agency](https://mlsanity.com/paper/2603.20633/) (2026)
- [VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models](https://mlsanity.com/paper/2603.06148/) (2026)
- [PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments](https://mlsanity.com/paper/2603.23231/) (2026)
- [Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agents](https://mlsanity.com/paper/2512.20092/) (2025)
- [MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigm](https://mlsanity.com/paper/2512.02895/) (2025)
- [Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects](https://mlsanity.com/paper/2604.05546/) (2026)
- [A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends](https://mlsanity.com/paper/2507.09861/) (2025)
- [SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models](https://mlsanity.com/paper/2512.05955/) (2025)
- [MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models](https://mlsanity.com/paper/2605.14906/) (2026)
- [The Cognitive Bandwidth Bottleneck: Shifting Long-Horizon Agent from Planning with Actions to Planning with Schemas](https://mlsanity.com/paper/2510.07091/) (2025)
- [Tactile Modality Fusion for Vision-Language-Action Models](https://mlsanity.com/paper/2603.14604/) (2026)
- [Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads](https://mlsanity.com/paper/2606.05843/) (2026)
- [FC-MIR: A Mobile Screen Awareness Framework for Intent-Aware Recommendation based on Frame-Compressed Multimodal Trajectory Reasoning](https://mlsanity.com/paper/2512.19107/) (2025)
- [How to Train Your Long-Context Visual Document Model](https://mlsanity.com/paper/2602.15257/) (2026)
- [Hierarchical Evidence-Driven Reasoning for Long Document Understanding](https://mlsanity.com/paper/2607.04625/) (2026)
- [Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judge](https://mlsanity.com/paper/2602.11340/) (2026)
- [Fix Before Search: Benchmarking Agentic Query Visual Pre-processing in Multimodal Retrieval-augmented Generation](https://mlsanity.com/paper/2602.13179/) (2026)
- [MMCL-Bench: Multimodal Context Learning from Visual Rules, Procedures, and Evidence](https://mlsanity.com/paper/2605.12703/) (2026)
- [Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context](https://mlsanity.com/paper/2605.13831/) (2026)
- [Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models](https://mlsanity.com/paper/2605.27243/) (2026)
- [CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition](https://mlsanity.com/paper/2607.25294/) (2026)
- [PMMC: Prospective Multimodal Memory Compilation for Long-Term LVLM Agents](https://mlsanity.com/paper/2608.00962/) (2026)
- [MMLongCite: A Benchmark for Evaluating Fidelity of Long-Context Vision-Language Models](https://mlsanity.com/paper/2510.13276/) (2025)
- [Internalized Reasoning for Long-Context Visual Document Understanding](https://mlsanity.com/paper/2604.02371/) (2026)
- [MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios](https://mlsanity.com/paper/2606.14747/) (2026)
- [SciLens: Multi-modal Scientific Claim Verification with Agentic Entailment and Grounding](https://mlsanity.com/paper/2606.20873/) (2026)
- [How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations](https://mlsanity.com/paper/2606.26041/) (2026)
- [SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding](https://mlsanity.com/paper/2607.10400/) (2026)
- [VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling](https://mlsanity.com/paper/2608.08630/) (2026)

## Abstract (arXiv)

The rapid extension of context windows in large vision-language models has given rise to long-context vision-language models (LCVLMs), which are capable of handling hundreds of images with interleaved text tokens in a single forward pass. In this work, we introduce MMLongBench, the first benchmark covering a diverse set of long-context vision-language tasks, to evaluate LCVLMs effectively and thoroughly. MMLongBench is composed of 13,331 examples spanning five different categories of downstream tasks, such as Visual RAG and Many-Shot ICL. It also provides broad coverage of image types, including various natural and synthetic images. To assess the robustness of the models to different input lengths, all examples are delivered at five standardized input lengths (8K-128K tokens) via a cross-modal tokenization scheme that combines vision patches and text tokens. Through a thorough benchmarking of 46 closed-source and open-source LCVLMs, we provide a comprehensive analysis of the current models' vision-language long-context ability. Our results show that: i) performance on a single task is a weak proxy for overall long-context capability; ii) both closed-source and open-source models face challenges in long-context vision-language tasks, indicating substantial room for future improvement; iii) models with stronger reasoning ability tend to exhibit better long-context performance. By offering wide task coverage, various image types, and rigorous length control, MMLongBench provides the missing foundation for diagnosing and advancing the next generation of LCVLMs.