mlsanityData to 14 Oct 2026

SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension

arXiv 2307.16125 ↗First posted 30 Jul 2023cs.CL cs.CVcited by 940

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Based on powerful Large Language Models (LLMs), recent generative Multimodal Large Language Models (MLLMs) have gained prominence as a pivotal research area, exhibiting remarkable capability for both comprehension and generation. In this work, we address the evaluation of generative comprehension in MLLMs as a preliminary step towards a comprehensive assessment of generative models, by introducing a benchmark named SEED-Bench. SEED-Bench consists of 19K multiple choice questions with accurate human annotations (x 6 larger than existing benchmarks), which spans 12 evaluation dimensions including the comprehension of both the image and video modality. We develop an advanced pipeline for generating multiple-choice questions that target specific evaluation dimensions, integrating both automatic filtering and manual verification processes. Multiple-choice questions with groundtruth options derived from human annotation enables an objective and efficient assessment of model performance, eliminating the need for human or GPT intervention during evaluation. We further evaluate the performance of 18 models across all 12 dimensions, covering both the spatial and temporal understanding. By revealing the limitations of existing MLLMs through evaluation results, we aim for SEED-Bench to provide insights for motivating future research. We will launch and consistently maintain a leaderboard to provide a platform for the community to assess and investigate model capability.

Key facts. Cited by 940 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 38 references that are in the map. First posted on arXiv 30 Jul 2023.

Builds on

30 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  3. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  4. 2023Generating Images with Multimodal Language Models26 May
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
  7. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
11 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  2. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  4. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  5. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  6. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  7. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  8. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  9. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 121
  10. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 86
  11. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 67; 2024: 340; 2025: 341; 2026: 192.

940 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  10. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  12. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
38 more
  1. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  4. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  5. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  6. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  7. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  8. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  9. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  10. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  11. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  12. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  13. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  14. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  15. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  16. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  17. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  18. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  19. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  20. 2024What matters when building vision-language models?cited by 326
  21. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  22. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  23. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  24. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  25. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  26. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  27. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  28. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  29. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 286
  30. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  31. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  32. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 251
  33. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  34. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  35. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  36. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  37. 2025Transfer between Modalities with MetaQueriescited by 236
  38. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232

50 most cited of 940 All 940 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  2. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models
  3. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models
  4. 2024LLMs Meet Multimodal Generation and Editing: A Survey
  5. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
  6. 2024Rethinking Generative Large Language Model Evaluation for Semantic Comprehension
  7. 2023On the Performance of Multimodal Language Models
  8. 2025MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams