mlsanityData to 7 Oct 2026

MileBench: Benchmarking MLLMs in Long Context

arXiv 2404.18532 ↗First posted 29 Apr 2024cs.CL cs.AI cs.CVcited by 86

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Despite the advancements and impressive performance of Multimodal Large Language Models (MLLMs) on benchmarks, their effectiveness in real-world, long-context, and multi-image tasks is unclear due to the benchmarks' limited scope. Existing benchmarks often focus on single-image and short-text samples, and when assessing multi-image tasks, they either limit the image count or focus on specific task (e.g time-series captioning), potentially obscuring the performance challenges of MLLMs. To address these limitations, we introduce MileBench, a pioneering benchmark designed to test the MultImodal Long-contExt capabilities of MLLMs. This benchmark comprises not only multimodal long contexts, but also multiple tasks requiring both comprehension and generation. We establish two distinct evaluation sets, diagnostic and realistic, to systematically assess MLLMs' long-context adaptation capacity and their ability to complete tasks in long-context scenarios. Our experimental results, obtained from testing 22 models, revealed that while the closed-source GPT-4o outperforms others, most open-source MLLMs struggle in long-context situations. Interestingly, the performance gap tends to widen with an increase in the number of images. We strongly encourage an intensification of research efforts towards enhancing MLLMs' long-context capabilities, especially in scenarios involving multiple images.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 56 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

40 of 56 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  2. 2023VILA: On Pre-training for Visual Language Models12 Dec
  3. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models28 Nov
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2024Yi: Open Foundation Models by 01.AIcited by 706
  11. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  12. 2017TALL: Temporal Activity Localization via Language Querycited by 581
19 more
  1. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  2. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  3. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  4. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  5. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  6. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  7. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  8. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  9. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  10. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  11. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  12. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  13. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  14. 2021WebQA: Multihop and Multimodal QAcited by 109
  15. 2019Expressing Visual Relationships via Languagecited by 49
  16. 2020ManyModalQA: Modality Disambiguation and QA over Diverse Inputscited by 38
  17. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  18. 2023An Early Evaluation of GPT-4V(ision)cited by 31
  19. 2021GPR1200: A Benchmark for General-Purpose Content-Based Image Retrievalcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 31; 2026: 27.

86 papers.

Sort:

By area:

  1. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  3. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  4. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  5. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  6. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  7. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  8. 2024Needle In A Multimodal Haystackcited by 56
  9. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  10. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53
  11. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMscited by 47
  12. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
38 more
  1. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  2. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  3. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Modelscited by 34
  4. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  5. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  6. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  7. 2024ReMI: A Dataset for Reasoning with Multiple Imagescited by 27
  8. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  9. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  10. 2024LOVA3: Learning to Visual Question Answering, Asking and Assessmentcited by 21
  11. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmarkcited by 20
  12. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 15
  13. 2024Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contextscited by 14
  14. 2025MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inferencecited by 12
  15. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  16. 2024Cross-Self KV Cache Pruning for Efficient Vision-Language Inferencecited by 11
  17. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  18. 2025Abacus: A Cost-Based Optimizer for Semantic Operator Systemscited by 11
  19. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agentscited by 11
  20. 2024OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understandingcited by 10
  21. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10
  22. 2024Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMcited by 9
  23. 2025AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Modelscited by 9
  24. 2025From Easy to Hard: The MIR Benchmark for Progressive Interleaved Multi-Image Reasoningcited by 8
  25. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  26. 2024Benchmarking General-Purpose In-Context Learningcited by 6
  27. 2025Rethinking Causal Mask Attention for Vision-Language Inferencecited by 6
  28. 2025Draft-based Approximate Inference for LLMscited by 6
  29. 2025Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmarkcited by 6
  30. 2025FlowMM: Cross-Modal Information Flow Guided KV Cache Merging for Efficient Multimodal Context Inferencecited by 6
  31. 2024Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidencecited by 5
  32. 2026SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memorycited by 5
  33. 2024The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMscited by 4
  34. 2025Retrieval-augmented in-context learning for multimodal large language models in disease classificationcited by 4
  35. 2025Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?cited by 4
  36. 2025RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answeringcited by 4
  37. 2024Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMscited by 3
  38. 2025HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding?cited by 3

50 most cited of 86 All 86 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  2. 2024A Survey on Benchmarks of Multimodal Large Language Models
  3. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images
  4. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  5. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension
  6. 2025Benchmarking Large and Small MLLMs
  7. 2024The Revolution of Multimodal Large Language Models: A Survey
  8. 2024Can MLLMs Perform Text-to-Image In-Context Learning?