mlsanityData to 7 Oct 2026

TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens

arXiv 2410.05261 ↗First posted 7 Oct 2024cs.CV cs.AIcited by 32

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Reading dense text and locating objects within images are fundamental abilities for Large Vision-Language Models (LVLMs) tasked with advanced jobs. Previous LVLMs, including superior proprietary models like GPT-4o, have struggled to excel in both tasks simultaneously. Moreover, previous LVLMs with fine-grained perception cost thousands of tokens per image, making them resource-intensive. We present TextHawk2, a bilingual LVLM featuring efficient fine-grained perception and demonstrating cutting-edge performance across general-purpose, OCR, and grounding tasks with 16 times fewer image tokens. Critical improvements include: (1) Token Compression: Building on the efficient architecture of its predecessor, TextHawk2 significantly reduces the number of tokens per image by 16 times, facilitating training and deployment of the TextHawk series with minimal resources. (2) Visual Encoder Reinforcement: We enhance the visual encoder through LVLM co-training, unlocking its potential for previously unseen tasks like Chinese OCR and grounding. (3) Data Diversity: We maintain a comparable scale of 100 million samples while diversifying the sources of pre-training data. We assess TextHawk2 across multiple benchmarks, where it consistently delivers superior performance and outperforms closed-source models of similar scale, such as achieving 78.4% accuracy on OCRBench, 81.4% accuracy on ChartQA, 89.6% ANLS on DocVQA, and 88.1% [email protected] on RefCOCOg-test.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 105 of its 132 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

105 of 132 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  2. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images18 Mar

Other references · 103

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
38 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  12. 2021Grounded Language-Image Pre-trainingcited by 1,147
  13. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  14. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  15. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  16. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  17. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  18. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  19. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  20. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  21. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  22. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  23. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  24. 2016A Diagram Is Worth A Dozen Imagescited by 793
  25. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  26. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  27. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  28. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  29. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  30. 2021ImageNet-21K Pretraining for the Massescited by 607
  31. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  32. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  33. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  34. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  35. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  36. 2021InfographicVQAcited by 430
  37. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  38. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 16; 2026: 9.

32 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  5. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 78
  6. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
  7. 2025Ocean-OCR: Towards General OCR Application via a Vision-Language Modelcited by 24
  8. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compressioncited by 21
  9. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  10. 2025Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMscited by 20
  11. 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Promptingcited by 20
  12. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
20 more
  1. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  2. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  3. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  4. 2025VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoningcited by 7
  5. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  6. 2025Multi-Agent Interactive Question Generation Framework for Long Document Understandingcited by 4
  7. 2026Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligencecited by 4
  8. 2025DocTron-Formula: Generalized Formula Recognition in Complex and Structured Scenarioscited by 2
  9. 2024Temporal Contrastive Learning for Video Temporal Reasoning in Large Vision-Language Modelscited by 1
  10. 2025VaPR -- Vision-language Preference alignment for Reasoningcited by 1
  11. 2026Q-Mask: Query-driven Causal Masks for Text Anchoring in OCR-Oriented Vision-Language Modelscited by 1
  12. 2026MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systemscited by 1
  13. 2024Dynamic Cross-Modal Alignment for Robust Semantic Location Predictioncited by 0
  14. 2024Selective State Space Memory for Large Vision-Language Modelscited by 0
  15. 2024Optimizing Vision-Language Interactions Through Decoder-Only Modelscited by 0
  16. 2025Vision-Driven Prompt Optimization for Large Language Models in Multimodal Generative Taskscited by 0
  17. 2026DocCogito: Aligning Layout Cognition and Step-Level Grounded Reasoning for Document Understandingcited by 0
  18. 2026AdaThinking-E: One-Token Entropy Regulation for Adaptive Thinkingcited by 0
  19. 2026SupGRPO: Enhancing GRPO with Matching-based Online SFT for Text Spottingcited by 0
  20. 2026Xiaomi-OCR-0 Technical Reportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models
  2. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  3. 2024[CLS] Token Tells Everything Needed for Training-free Efficient MLLMs
  4. 2026Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models
  5. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  6. 2024iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models
  7. 2025Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning
  8. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis