mlsanityData to 7 Oct 2026

SEED-Bench-2: Benchmarking Multimodal Large Language Models

arXiv 2311.17092 ↗First posted 28 Nov 2023cs.CVcited by 96

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs), building upon the foundation of powerful large language models (LLMs), have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs (acting like a combination of GPT-4V and DALL-E 3). However, existing MLLM benchmarks remain limited to assessing only models' comprehension ability of single image-text inputs, failing to keep up with the strides made in MLLMs. A comprehensive benchmark is imperative for investigating the progress and uncovering the limitations of current MLLMs. In this work, we categorize the capabilities of MLLMs into hierarchical levels from L0 to L4 based on the modalities they can accept and generate, and propose SEED-Bench-2, a comprehensive benchmark that evaluates the hierarchical capabilities of MLLMs. Specifically, SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions, including the evaluation of both text and image generation. Multiple-choice questions with groundtruth options derived from human annotation enables an objective and efficient assessment of model performance, eliminating the need for human or GPT intervention during evaluation. We further evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations. By revealing the limitations of existing MLLMs through extensive evaluations, we aim for SEED-Bench-2 to provide insights that will motivate future research towards the goal of General Artificial Intelligence. Dataset and evaluation code are available at hrefhttps://github.com/AILab-CVC/SEED-Bench

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 66 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

52 of 66 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  3. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  4. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  5. 2023TouchStone: Evaluating Vision-Language Models by Language Models31 Aug
  6. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  11. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
2 more
  1. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
  2. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
26 more
  1. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  6. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  7. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  8. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  9. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  10. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  11. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  12. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  13. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  14. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  15. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  16. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  17. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  18. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  19. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  20. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  21. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  22. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  23. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  24. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 84
  25. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 66
  26. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 58; 2025: 22; 2026: 14.

96 papers.

Sort:

By area:

  1. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  2. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  3. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  4. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  5. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  6. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  7. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  8. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  9. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 133
  10. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  11. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  12. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
38 more
  1. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  2. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 75
  3. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  4. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  5. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  6. 2024TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domainscited by 68
  7. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 66
  8. 2024Needle In A Multimodal Haystackcited by 56
  9. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  10. 2023LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videoscited by 51
  11. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 50
  12. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 47
  13. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 41
  14. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  15. 2024Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMscited by 35
  16. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 35
  17. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 32
  18. 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoningcited by 31
  19. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  20. 2024SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Modelscited by 28
  21. 2024Safety Alignment for Vision Language Modelscited by 27
  22. 2024ReMI: A Dataset for Reasoning with Multiple Imagescited by 27
  23. 2024Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Modelscited by 27
  24. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  25. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  26. 2025SITE: towards Spatial Intelligence Thorough Evaluationcited by 26
  27. 2025MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Modelscited by 24
  28. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMscited by 21
  29. 2024Multimodal Misinformation Detection using Large Vision-Language Modelscited by 21
  30. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  31. 2024Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plotscited by 20
  32. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 19
  33. 2024II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Modelscited by 15
  34. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
  35. 2024Can MLLMs Understand the Deep Implication Behind Chinese Images?cited by 14
  36. 2024LATTE: Learning to Think with Vision Specialistscited by 13
  37. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  38. 2025VisNumBench: Evaluating Number Sense of Multimodal Large Language Modelscited by 12

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Benchmarks of Multimodal Large Language Models
  2. 2024The Revolution of Multimodal Large Language Models: A Survey
  3. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  4. 2024Efficient Multimodal Large Language Models: A Survey
  5. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images
  6. 2024LIME: Less Is More for MLLM Evaluation
  7. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  8. 2025Benchmarking Large and Small MLLMs