mlsanityData to 7 Oct 2026

MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmark

arXiv 2410.11538 ↗First posted 15 Oct 2024cs.CVcited by 26

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The comprehension of text-rich visual scenes has become a focal point for evaluating Multi-modal Large Language Models (MLLMs) due to their widespread applications. Current benchmarks tailored to the scenario emphasize perceptual capabilities, while overlooking the assessment of cognitive abilities. To address this limitation, we introduce a Multimodal benchmark towards Text-rich visual scenes, to evaluate the Cognitive capabilities of MLLMs through visual reasoning and content-creation tasks (MCTBench). To mitigate potential evaluation bias from the varying distributions of datasets, MCTBench incorporates several perception tasks (e.g., scene text recognition) to ensure a consistent comparison of both the cognitive and perceptual capabilities of MLLMs. To improve the efficiency and fairness of content-creation evaluation, we conduct an automatic evaluation pipeline. Evaluations of various MLLMs on MCTBench reveal that, despite their impressive perceptual capabilities, their cognition abilities require enhancement. We hope MCTBench will offer the community an efficient resource to explore and enhance cognitive capabilities towards text-rich visual scenes.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 49 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

30 of 49 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension25 Apr
  2. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  3. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  4. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  6. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  7. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding19 Aug
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 20

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  11. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  12. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
8 more
  1. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  2. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  3. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  4. 2017Automatic Understanding of Image and Video Advertisementscited by 67
  5. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  6. 2024ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Modelscited by 19
  7. 2024XOV-Action: Towards Generalizable Open-Vocabulary Action Recognitioncited by 15
  8. 2021FoodLogoDet-1500: A Dataset for Large-Scale Food Logo Detection via Multi-Scale Feature Decoupling Networkcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 21.

26 papers.

Sort:

By area:

  1. 2025WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?cited by 26
  2. 2025Task-Oriented 6-DoF Grasp Pose Detection in Clutterscited by 4
  3. 2025TDRI: Two-Phase Dialogue Refinement and Co-Adaptation for Interactive Image Generationcited by 1
  4. 2025Cross-Modal Synergies: Unveiling the Potential of Motion-Aware Fusion Networks in Handling Dynamic and Static ReID Scenarioscited by 0
  5. 2025OMR-Diffusion:Optimizing Multi-Round Enhanced Training in Diffusion Models for Improved Intent Understandingcited by 0
  6. 2026LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Datacited by 0
  7. 2026Knowledge-Embedded and Hypernetwork-Guided Few-Shot Substation Meter Defect Image Generation Methodcited by 0
  8. 2026MA-LipNet: Multi-Dimensional Attention Networks for Robust Lipreadingcited by 0
  9. 2026The Diffusion Duet: Harmonizing Dual Channels with Wavelet Suppression for Image Separationcited by 0
  10. 2026Detecting Deepfakes with Multivariate Soft Blending and CLIP-based Image-Text Alignmentcited by 0
  11. 2026Efficient Image Super-Resolution with Multi-Scale Spatial Adaptive Attention Networkscited by 0
  12. 2026A Deformable Attention-Based Detection Transformer with Cross-Scale Feature Fusion for Industrial Coil Spring Inspectioncited by 0
14 more
  1. 2026JND-Guided Neural Watermarking with Spatial Transformer Decoding for Screen-Capture Robustnesscited by 0
  2. 2026Transmittance-Guided Structure-Texture Decomposition for Nighttime Image Dehazingcited by 0
  3. 2026Hierarchical Awareness Adapters with Hybrid Pyramid Feature Fusion for Dense Depth Predictioncited by 0
  4. 2026Adaptive Slicing-Assisted Hyper Inference for Enhanced Small Object Detection in High-Resolution Imagerycited by 0
  5. 2026Feature Perturbation Pool-based Fusion Network for Unified Multi-Class Industrial Defect Detectioncited by 0
  6. 2026Lightweight Real-Time Rendering Parameter Optimization via XGBoost-Driven Lookup Tablescited by 0
  7. 2026Image Classification via Random Dilated Convolution with Multi-Branch Feature Extraction and Context Excitationcited by 0
  8. 2026Gait Recognition via Deep Residual Networks and Multi-Branch Feature Fusioncited by 0
  9. 2026Multi-Branch Non-Homogeneous Image Dehazing via Concentration Partitioning and Image Fusioncited by 0
  10. 2026Local Spatiotemporal Convolutional Network for Robust Gait Recognitioncited by 0
  11. 2026Do You Need Text Rectification? Soft Attention Mask Embedding for Rectification-Free Scene Text Spottingcited by 0
  12. 2026DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoningcited by 0
  13. 2026ProWAFT: A ROMA-LPD Instance for Workload-Aware and Dynamic Fault Tolerance in FPGA-Based CNN Acceleratorscited by 0
  14. 2026Rank-Consistent Set Reasoning for Co-Salient Object Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  2. 2026WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark
  3. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning
  4. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
  5. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  6. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  7. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
  8. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images