mlsanityData to 7 Oct 2026

LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models

arXiv 2407.12772 ↗First posted 17 Jul 2024cs.CL cs.CVcited by 344

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The advances of large foundation models necessitate wide-coverage, low-cost, and zero-contamination benchmarks. Despite continuous exploration of language model evaluations, comprehensive studies on the evaluation of Large Multi-modal Models (LMMs) remain limited. In this work, we introduce LMMS-EVAL, a unified and standardized multimodal benchmark framework with over 50 tasks and more than 10 models to promote transparent and reproducible evaluations. Although LMMS-EVAL offers comprehensive coverage, we find it still falls short in achieving low cost and zero contamination. To approach this evaluation trilemma, we further introduce LMMS-EVAL LITE, a pruned evaluation toolkit that emphasizes both coverage and efficiency. Additionally, we present Multimodal LIVEBENCH that utilizes continuously updating news and online forums to assess models' generalization abilities in the wild, featuring a low-cost and zero-contamination evaluation approach. In summary, our work highlights the importance of considering the evaluation trilemma and provides practical solutions to navigate the trade-offs in evaluating large multi-modal models, paving the way for more effective and reliable benchmarking of LMMs. We opensource our codebase and maintain leaderboard of LIVEBENCH at https://github.com/EvolvingLMMs-Lab/lmms-eval and https://huggingface.co/spaces/lmms-lab/LiveBench.

Key facts. Cited by 344 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 93 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

56 of 93 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models3 May
  2. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models18 Apr
  3. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  4. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark22 Jan
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  6. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models5 Dec
  7. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  8. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages23 Aug
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  10. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
1 more
  1. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
31 more
  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  3. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  7. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  8. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2016A Diagram Is Worth A Dozen Imagescited by 793
  12. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  13. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  14. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  15. 2018nocaps: novel object captioning at scalecited by 444
  16. 2021InfographicVQAcited by 430
  17. 2021OCR-free Document Understanding Transformercited by 400
  18. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  19. 2019Scene Text Visual Question Answeringcited by 327
  20. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  21. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  22. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  23. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  24. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  25. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  26. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  27. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  28. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  29. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  30. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  31. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 36; 2025: 169; 2026: 139.

344 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  5. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  6. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  7. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  8. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  9. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  10. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  11. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  12. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
38 more
  1. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  2. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  3. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  4. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  5. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
  6. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  7. 2025MiMo-VL Technical Reportcited by 106
  8. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  9. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Modelscited by 92
  10. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  11. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  12. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  13. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  14. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 72
  15. 2025VGR: Visual Grounded Reasoningcited by 70
  16. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Callingcited by 58
  17. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Modelcited by 57
  18. 2025FineVision: Open Data Is All You Needcited by 51
  19. 2025XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery?cited by 50
  20. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  21. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 47
  22. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  23. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 45
  24. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Modelcited by 44
  25. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  26. 2025BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulationcited by 42
  27. 2025GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolutioncited by 41
  28. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 41
  29. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  30. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  31. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 38
  32. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  33. 2025OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipecited by 37
  34. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 34
  35. 2025Unhackable Temporal Rewarding for Scalable Video MLLMscited by 33
  36. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  37. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  38. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31

50 most cited of 344 All 344 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Large Multimodal Models against Common Corruptions
  2. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  3. 2024Automatic benchmarking of large multimodal models via iterative experiment programming
  4. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning
  5. 2026What We are Missing in Multimodal LLM Evaluation?
  6. 2024MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks
  7. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Models
  8. 2025LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs