mlsanityData to 14 Oct 2026

TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Models

arXiv 2404.09204 ↗First posted 14 Apr 2024cs.CV cs.AIcited by 27

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have shown impressive results on various multimodal tasks. However, most existing MLLMs are not well suited for document-oriented tasks, which require fine-grained image perception and information compression. In this paper, we present TextHawk, a MLLM that is specifically designed for document-oriented tasks, while preserving the general capabilities of MLLMs. TextHawk is aimed to explore efficient fine-grained perception by designing four dedicated components. Firstly, a ReSampling and ReArrangement (ReSA) module is proposed to reduce the redundancy in the document texts and lower the computational cost of the MLLM. We explore encoding the positions of each local feature by presenting Scalable Positional Embeddings (SPEs), which can preserve the scalability of various image sizes. A Query Proposal Network (QPN) is then adopted to initialize the queries dynamically among different sub-images. To further enhance the fine-grained visual perceptual ability of the MLLM, we design a Multi-Level Cross-Attention (MLCA) mechanism that captures the hierarchical structure and semantic relations of document images. Furthermore, we create a new instruction-tuning dataset for document-oriented tasks by enriching the multimodal document data with Gemini Pro. We conduct extensive experiments on both general and document-oriented MLLM benchmarks, and show that TextHawk outperforms the state-of-the-art methods, demonstrating its effectiveness and superiority in fine-grained document perception and general abilities.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 51 references that are in the map. First posted on arXiv 14 Apr 2024.

Builds on

29 of 51 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  2. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  3. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  4. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding19 Aug
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  12. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
9 more
  1. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  2. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  3. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  4. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 179
  5. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  6. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  7. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 151
  8. 2020Point and Ask: Incorporating Pointing into Visual Question Answeringcited by 39
  9. 2024Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysiscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 6; 2026: 10.

27 papers.

Sort:

By area:

  1. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  2. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  3. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  4. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  5. 2024Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramidcited by 26
  6. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to Seecited by 23
  7. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compressioncited by 22
  8. 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Promptingcited by 22
  9. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  10. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  11. 2024PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Samplingcited by 14
  12. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 14
15 more
  1. 2024VMAD: Visual-enhanced Multimodal Large Language Model for Zero-Shot Anomaly Detectioncited by 13
  2. 2025Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Modelscited by 11
  3. 2025VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoningcited by 7
  4. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 7
  5. 2025Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiencycited by 5
  6. 2026Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Modelscited by 4
  7. 2025Learning Compact Vision Tokens for Efficient Large Multimodal Modelscited by 3
  8. 2026Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplificationcited by 3
  9. 2026ReAlign: Optimizing the Visual Document Retriever with Reasoning-Guided Fine-Grained Alignmentcited by 3
  10. 2026Grounded 3D-Aware Spatial Vision-Language Modelingcited by 3
  11. 2026Geometric Risk Control for Vision-Language Model OCRcited by 1
  12. 2026MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systemscited by 1
  13. 2024Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspectivecited by 0
  14. 2026PositionOCR: Augmenting Positional Awareness in Multi-Modal Models via Hybrid Specialist Integrationcited by 0
  15. 2026Detached Skip-Links and R-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Revolution of Multimodal Large Language Models: A Survey
  2. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  3. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming
  4. 2025Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models
  5. 2024From Text to Pixel: Advancing Long-Context Understanding in MLLMs
  6. 2024A Survey on Benchmarks of Multimodal Large Language Models
  7. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
  8. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models