mlsanityData to 14 Oct 2026

TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding

arXiv 2404.09797 ↗First posted 15 Apr 2024cs.CVcited by 38

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

The advent of Large Multimodal Models (LMMs) has sparked a surge in research aimed at harnessing their remarkable reasoning abilities. However, for understanding text-rich images, challenges persist in fully leveraging the potential of LMMs, and existing methods struggle with effectively processing high-resolution images. In this work, we propose TextCoT, a novel Chain-of-Thought framework for text-rich image understanding. TextCoT utilizes the captioning ability of LMMs to grasp the global context of the image and the grounding capability to examine local textual regions. This allows for the extraction of both global and local visual information, facilitating more accurate question-answering. Technically, TextCoT consists of three stages, including image overview, coarse localization, and fine-grained observation. The image overview stage provides a comprehensive understanding of the global scene information, and the coarse localization stage approximates the image area containing the answer based on the question asked. Then, integrating the obtained global image descriptions, the final stage further examines specific regions to provide accurate answers. Our method is free of extra training, offering immediate plug-and-play functionality. Extensive experiments are conducted on a series of text-rich image question-answering benchmark datasets based on several advanced LMMs, and the results demonstrate the effectiveness and strong generalization ability of our method. Code is available at https://github.com/bzluan/TextCoT.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 69 of its 85 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

69 of 85 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs5 Jan
  2. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
  3. 2023Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts15 Nov
  4. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models25 Oct
  5. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  9. 2019Towards VQA Models That Can Readcited by 1,718
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  13. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  14. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 991
  15. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  16. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  17. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  18. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  19. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  20. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  21. 2021InfographicVQAcited by 437
  22. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  23. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  24. 2019Scene Text Visual Question Answeringcited by 331
  25. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  26. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  27. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  28. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  29. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  30. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  31. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  32. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 151
  33. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  34. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  35. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  36. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 95
  37. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  38. 2023VIGC: Visual Instruction Generation and Correctioncited by 80

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 20; 2026: 12.

38 papers.

Sort:

By area:

  1. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  3. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 175
  4. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
  5. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 136
  6. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  7. 2026Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillationcited by 77
  8. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  9. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  10. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perceptioncited by 50
  11. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 46
  12. 2025VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learningcited by 43
26 more
  1. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 35
  2. 2025SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Modelscited by 29
  3. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  4. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  5. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understandingcited by 22
  6. 2025Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinementcited by 18
  7. 2025GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinkingcited by 15
  8. 2025CROP: Contextual Region-Oriented Visual Token Pruningcited by 14
  9. 2025Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmarkcited by 11
  10. 2025Reinforcing VLMs to Use Tools for Detailed Visual Reasoning Under Resource Constraintscited by 6
  11. 2026M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understandingcited by 6
  12. 2025Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMscited by 4
  13. 2024Image First or Text First? Optimising the Sequencing of Modalities in Large Language Model Prompting and Reasoning Taskscited by 3
  14. 2024CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Modelscited by 3
  15. 2025Training-Free Reasoning and Reflection in MLLMscited by 3
  16. 2026Global-Local Dual Perception for MLLMs in High-Resolution Text-Rich Image Translationcited by 3
  17. 2026HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Frameworkcited by 3
  18. 2026Lyapunov Probes for Hallucination Detection in Large Foundation Modelscited by 3
  19. 2026Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Rewardcited by 2
  20. 2025Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoningcited by 1
  21. 2025MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understandingcited by 1
  22. 2026MoRL: Reinforced Reasoning for Unified Motion Understanding and Generationcited by 0
  23. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  24. 2026Question-Guided Evidence Acquisition for Multimodal Visual Question Answeringcited by 0
  25. 2026First Things First: Teaching LLM-Based Agents to Prioritize Must-Haves before Nice-to-Havescited by 0
  26. 2026Benchmarking Multimodal Large Language Models for Zero-Shot Human-Object Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  2. 2024MMR: Evaluating Reading Ability of Large Multimodal Models
  3. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks
  4. 2024ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models
  5. 2025RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks
  6. 2024Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
  7. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models
  8. 2025A Simple Data Augmentation Strategy for Text-in-Image Scientific VQA