mlsanityData to 14 Oct 2026

MMR: Evaluating Reading Ability of Large Multimodal Models

arXiv 2408.14594 ↗First posted 26 Aug 2024cs.CVcited by 6

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large multimodal models (LMMs) have demonstrated impressive capabilities in understanding various types of image, including text-rich images. Most existing text-rich image benchmarks are simple extraction-based question answering, and many LMMs now easily achieve high scores. This means that current benchmarks fail to accurately reflect performance of different models, and a natural idea is to build a new benchmark to evaluate their complex reasoning and spatial understanding abilities. In this work, we propose the Multi-Modal Reading (MMR) benchmark in 11 diverse tasks to evaluate LMMs for text-rich image understanding. MMR is the first text-rich image benchmark built on human annotations with the help of language models. By evaluating several state-of-the-art LMMs, including GPT-4o, it reveals the limited capabilities of existing LMMs underscoring the value of our benchmark.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 54 references that are in the map. First posted on arXiv 26 Aug 2024.

Builds on

38 of 54 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  2. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
  3. 2024TRINS: Towards Multimodal Language Models that Can Read10 Jun
  4. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation14 May
  5. 2024Exploring the Capabilities of Large Multimodal Models on Dense Text9 May
  6. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  7. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  9. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  12. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorial26 Jun
2 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  2. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 24

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
12 more
  1. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  2. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  3. 2019Scene Text Visual Question Answeringcited by 331
  4. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  5. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 213
  6. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  7. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 171
  8. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 160
  9. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  10. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 100
  11. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  12. 2019ICDAR 2019 Competition on Scene Text Visual Question Answeringcited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  2. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 14
  3. 2025MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Modelscited by 4
  4. 2025Socratic Chart: Cooperating Multiple Agents for Robust SVG Chart Understandingcited by 3
  5. 2026DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agentscited by 3
  6. 2025PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Languagecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning
  2. 2025LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs
  3. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models
  4. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  5. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models
  6. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  7. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
  8. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities