mlsanityData to 14 Oct 2026

SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension

arXiv 2404.16790 ↗First posted 25 Apr 2024cs.CVcited by 146

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Comprehending text-rich visual content is paramount for the practical application of Multimodal Large Language Models (MLLMs), since text-rich scenarios are ubiquitous in the real world, which are characterized by the presence of extensive texts embedded within images. Recently, the advent of MLLMs with impressive versatility has raised the bar for what we can expect from MLLMs. However, their proficiency in text-rich scenarios has yet to be comprehensively and objectively assessed, since current MLLM benchmarks primarily focus on evaluating general visual comprehension. In this work, we introduce SEED-Bench-2-Plus, a benchmark specifically designed for evaluating text-rich visual comprehension of MLLMs. Our benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs, each of which covers a wide spectrum of text-rich scenarios in the real world. These categories, due to their inherent complexity and diversity, effectively simulate real-world text-rich environments. We further conduct a thorough evaluation involving 34 prominent MLLMs (including GPT-4V, Gemini-Pro-Vision and Claude-3-Opus) and emphasize the current limitations of MLLMs in text-rich visual comprehension. We hope that our work can serve as a valuable addition to existing MLLM benchmarks, providing insightful observations and inspiring further research in the area of text-rich visual comprehension with MLLMs. The dataset and evaluation code can be accessed at https://github.com/AILab-CVC/SEED-Bench.

Key facts. Cited by 146 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 53 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

46 of 53 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  2. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  3. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models28 Nov
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  7. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  8. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep
  9. 2023TouchStone: Evaluating Vision-Language Models by Language Models31 Aug
  10. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
6 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  3. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
  6. 2023VPGTrans: Transfer Visual Prompt Generator across LLMs2 May

Other references · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
16 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  2. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  4. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  5. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  6. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  7. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  8. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  9. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  10. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  11. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  12. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  13. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  14. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  15. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42
  16. 2024ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 25; 2025: 77; 2026: 44.

146 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  4. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  5. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  6. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  7. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  8. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
  9. 2025MiMo-VL Technical Reportcited by 115
  10. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
  11. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  12. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
38 more
  1. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 68
  2. 2024Needle In A Multimodal Haystackcited by 58
  3. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  4. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 51
  5. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 47
  6. 2025BitVLA: 1-bit Vision-Language-Action Models for Robotics Manipulationcited by 43
  7. 2025MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agentscited by 42
  8. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 41
  9. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 41
  10. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 39
  11. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  12. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 33
  13. 2025Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spacescited by 33
  14. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  15. 2025FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluationcited by 30
  16. 2025SITE: towards Spatial Intelligence Thorough Evaluationcited by 29
  17. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 28
  18. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  19. 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 27
  20. 2024MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmarkcited by 26
  21. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
  22. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  23. 2025RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learningcited by 21
  24. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  25. 2025Redundancy Principles for MLLMs Benchmarkscited by 19
  26. 2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18
  27. 2026Forest Before Trees: Latent Superposition for Efficient Visual Reasoningcited by 18
  28. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 18
  29. 2024Phantom of Latent for Large Language and Vision Modelscited by 17
  30. 2025NVIDIA Nemotron Nano V2 VLcited by 17
  31. 2025Play to Generalize: Learning to Reason Through Game Playcited by 16
  32. 2025LanP: Rethinking the Impact of Language Priors in Large Vision-Language Modelscited by 14
  33. 2025Oasis: One Image is All You Need for Multimodal Instruction Data Synthesiscited by 14
  34. 2026Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Modelscited by 14
  35. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Modelscited by 13
  36. 2026Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusioncited by 13
  37. 2025Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Datacited by 12
  38. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learningcited by 11

50 most cited of 146 All 146 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Benchmarks of Multimodal Large Language Models
  2. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding
  3. 2025Evaluating Variance in Visual Question Answering Benchmarks
  4. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  5. 2024MileBench: Benchmarking MLLMs in Long Context
  6. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  7. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  8. 2024CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models