mlsanityData to 7 Oct 2026

LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models

arXiv 2306.09265 ↗First posted 15 Jun 2023cs.CV cs.AIcited by 217

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have recently played a dominant role in multimodal vision-language learning. Despite the great success, it lacks a holistic evaluation of their efficacy. This paper presents a comprehensive evaluation of publicly available large multimodal models by building a LVLM evaluation Hub (LVLM-eHub). Our LVLM-eHub consists of 8 representative LVLMs such as InstructBLIP and MiniGPT-4, which are thoroughly evaluated by a quantitative capability evaluation and an online arena platform. The former evaluates 6 categories of multimodal capabilities of LVLMs such as visual question answering and embodied artificial intelligence on 47 standard text-related visual benchmarks, while the latter provides the user-level evaluation of LVLMs in an open-world question-answering scenario. The study reveals several innovative findings. First, instruction-tuned LVLM with massive in-domain data such as InstructBLIP heavily overfits many existing tasks, generalizing poorly in the open-world scenario. Second, instruction-tuned LVLM with moderate instruction-following data may result in object hallucination issues (i.e., generate objects that are inconsistent with target images in the descriptions). It either makes the current evaluation metric such as CIDEr for image captioning ineffective or generates wrong answers. Third, employing a multi-turn reasoning evaluation framework can mitigate the issue of object hallucination, shedding light on developing an effective pipeline for LVLM evaluation. The findings provide a foundational framework for the conception and assessment of innovative strategies aimed at enhancing zero-shot multimodal techniques. Our LVLM-eHub will be available at https://github.com/OpenGVLab/Multi-Modality-Arena

Key facts. Cited by 217 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 72 of its 120 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

72 of 120 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  2. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
  3. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 69

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  13. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  14. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  15. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  16. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  17. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  18. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  19. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  20. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  21. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  22. 2023Instruction Tuning with GPT-4cited by 676
  23. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  24. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  25. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  26. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  27. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  28. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  29. 2018nocaps: novel object captioning at scalecited by 444
  30. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  31. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  32. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  33. 2016Visual Dialogcited by 377
  34. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  35. 2022Visual Spatial Reasoningcited by 288
  36. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  37. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  38. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 45; 2024: 86; 2025: 63; 2026: 23.

217 papers.

Sort:

By area:

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  5. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  9. 2023A Survey on Multimodal Large Language Modelscited by 898
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  11. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  12. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
38 more
  1. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  2. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  3. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  4. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  5. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  6. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  7. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  8. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  9. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  10. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  11. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  12. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  13. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  14. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  15. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  16. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  17. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  18. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 81
  19. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  20. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 76
  21. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  22. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 66
  23. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  24. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasetscited by 61
  25. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  26. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Modelscited by 59
  27. 2024Needle In A Multimodal Haystackcited by 56
  28. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 51
  29. 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selectioncited by 48
  30. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 47
  31. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  32. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  33. 2024PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathologycited by 43
  34. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  35. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Modelscited by 41
  36. 2023Visual Instruction Tuning with Polite Flamingocited by 38
  37. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  38. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 37

50 most cited of 217 All 217 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  2. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  3. 2024A Survey on Benchmarks of Multimodal Large Language Models
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?
  5. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  6. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models
  7. 2025EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models
  8. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks