mlsanityData to 7 Oct 2026

Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review

arXiv 2502.16586 ↗First posted 23 Feb 2025cs.CVcited by 8

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The recent emergence of Multi-modal Large Language Models (MLLMs) has introduced a new dimension to the Text-rich Image Understanding (TIU) field, with models demonstrating impressive and inspiring performance. However, their rapid evolution and widespread adoption have made it increasingly challenging to keep up with the latest advancements. To address this, we present a systematic and comprehensive survey to facilitate further research on TIU MLLMs. Initially, we outline the timeline, architecture, and pipeline of nearly all TIU MLLMs. Then, we review the performance of selected models on mainstream benchmarks. Finally, we explore promising directions, challenges, and limitations within the field.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 107 of its 147 references that are in the map. First posted on arXiv 23 Feb 2025.

Builds on

107 of 147 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  4. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning30 Sep
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  6. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders28 Aug
  7. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  9. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension25 Apr
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  11. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  12. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
3 more
  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding19 Aug
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 92

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Segment Anythingcited by 8,158
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2025Qwen2.5-VL Technical Reportcited by 5,418
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
38 more
  1. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  2. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  3. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  4. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  5. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  6. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  8. 2021InfographicVQAcited by 430
  9. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  10. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  11. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  12. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  13. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  14. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  15. 2019Scene Text Visual Question Answeringcited by 327
  16. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  17. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  18. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  19. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  20. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  21. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  22. 2019Image-based table recognition: data, model, and evaluationcited by 233
  23. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  24. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  25. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  26. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  27. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  28. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  29. 2017Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognitioncited by 161
  30. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  31. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  32. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  33. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  34. 2019ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVTcited by 121
  35. 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 120
  36. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  37. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  38. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  2. 2026CodePercept: Code-Grounded Visual STEM Perception for MLLMscited by 3
  3. 2025TextlessRAG: End-to-End Visual Document RAG by Speech Without Textcited by 2
  4. 2026TextRich: A Multi-Domain Benchmark for Detecting AI-Generated Text-Rich Images from GPT-Image-2cited by 2
  5. 2025Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoningcited by 1
  6. 2026LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinementcited by 1
  7. 2026VaRS-Doc: Interpretation-Aware Variant Representations via Latent Self-Probing for Visual Document Retrievalcited by 0
  8. 2026Xiaomi-OCR-0 Technical Reportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Revolution of Multimodal Large Language Models: A Survey
  2. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  3. 2024Efficient Multimodal Large Language Models: A Survey
  4. 2024From Text to Pixel: Advancing Long-Context Understanding in MLLMs
  5. 2023Multimodal Large Language Models: A Survey
  6. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey
  7. 2024Personalized Multimodal Large Language Models: A Survey
  8. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models