mlsanityData to 14 Oct 2026

VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models

arXiv 2407.11691 ↗First posted 16 Jul 2024cs.CVcited by 455

Not on the map. Why?

Cited by 307 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present VLMEvalKit: an open-source toolkit for evaluating large multi-modality models based on PyTorch. The toolkit aims to provide a user-friendly and comprehensive framework for researchers and developers to evaluate existing multi-modality models and publish reproducible evaluation results. In VLMEvalKit, we implement over 450+ large multi-modality model configurations, including both proprietary APIs and open-source models, and support 330+ benchmarks across diverse multi-modal benchmarks. By implementing a single interface, new models can be easily added to the toolkit, while the toolkit automatically handles the remaining workloads, including data preparation, distributed inference, prediction post-processing, and metric calculation. VLMEvalKit has also evolved to a broader evaluation suite spanning video/audio, document understanding, GUI grounding, spatial reasoning, safety, scientific reasoning, and multi-turn dialogue. Based on the evaluation results obtained with the toolkit, we host the OpenVLM Leaderboard, a comprehensive leaderboard to track the progress of multi-modality learning research. The toolkit is released on https://github.com/open-compass/VLMEvalKit and is actively maintained.

Key facts. Cited by 455 papers in the map (data through 14 Oct 2026). Builds on 91 of its 211 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

91 of 211 references.

References · 91

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  11. 2019Towards VQA Models That Can Readcited by 1,718
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
38 more
  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  3. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  13. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  14. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  15. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  16. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  17. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  18. 2016A Diagram Is Worth A Dozen Imagescited by 804
  19. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  20. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  21. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  22. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  23. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  24. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  25. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  26. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  27. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  28. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  29. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  30. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  31. 2021InfographicVQAcited by 437
  32. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  33. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  34. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  35. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  36. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  37. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  38. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 45; 2025: 224; 2026: 186.

455 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  3. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  4. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  5. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 303
  6. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  7. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  8. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 230
  9. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 212
  10. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 199
  11. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 183
  12. 2024Building and better understanding vision-language models: insights and future directionscited by 173
38 more
  1. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  2. 2025Thyme: Think Beyond Imagescited by 167
  3. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 161
  4. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  5. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 136
  6. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  7. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  8. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 125
  9. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
  10. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  11. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 113
  12. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  13. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  14. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 84
  15. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  16. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  17. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 77
  18. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  19. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  20. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 72
  21. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learningcited by 69
  22. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 69
  23. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Datacited by 67
  24. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  25. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 63
  26. 2024Reconstructive Visual Instruction Tuningcited by 55
  27. 2025Scalable Vision Language Model Training via High Quality Data Curationcited by 55
  28. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 53
  29. 2025Advancing Multimodal Reasoning: From Optimized Cold Start to Staged Reinforcement Learningcited by 53
  30. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 53
  31. 2025LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?cited by 49
  32. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  33. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encodingcited by 44
  34. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 43
  35. 2025CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generationcited by 43
  36. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  37. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 42
  38. 2025Vision Transformers Don't Need Trained Registerscited by 41

50 most cited of 455 All 455 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  2. 2024OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities
  3. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models
  4. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
  5. 2024OmniBench: Towards The Future of Universal Omni-Language Models
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  7. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
  8. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks