mlsanityData to 7 Oct 2026

A Token-level Text Image Foundation Model for Document Understanding

arXiv 2503.02304 ↗First posted 4 Mar 2025cs.CVcited by 13

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

In recent years, general visual foundation models (VFMs) have witnessed increasing adoption, particularly as image encoders for popular multi-modal large language models (MLLMs). However, without semantically fine-grained supervision, these models still encounter fundamental prediction errors in the context of downstream text-image-related tasks, i.e., perception, understanding and reasoning with images containing small and dense texts. To bridge this gap, we develop TokenOCR, the first token-level visual foundation model specifically tailored for text-image-related tasks, designed to support a variety of traditional downstream applications. To facilitate the pretraining of TokenOCR, we also devise a high-quality data production pipeline that constructs the first token-level image text dataset, TokenIT, comprising 20 million images and 1.8 billion token-mask pairs. Furthermore, leveraging this foundation with exceptional image-as-text capability, we seamlessly replace previous VFMs with TokenOCR to construct a document-level MLLM, TokenVL, for VQA-based document understanding tasks. Finally, extensive experiments demonstrate the effectiveness of TokenOCR and TokenVL. Code, datasets, and weights will be available at https://github.com/Token-family/TokenFD.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 107 of its 133 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

107 of 133 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding8 Nov
  2. 2024PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Sampling8 Oct
  3. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming27 Jun
  4. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models17 Jun
  5. 2024HRVDA: High-Resolution Visual Document Assistant10 Apr
  6. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding8 Apr
  7. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document7 Mar
  8. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding20 Nov

Other references · 99

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Segment Anythingcited by 8,158
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
38 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  9. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  10. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  11. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  12. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  13. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  14. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  15. 2021InfographicVQAcited by 430
  16. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  17. 2021OCR-free Document Understanding Transformercited by 400
  18. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  19. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  20. 2019PubLayNet: largest dataset ever for document layout analysiscited by 349
  21. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  22. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 345
  23. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  24. 2019Scene Text Visual Question Answeringcited by 327
  25. 2024What matters when building vision-language models?cited by 321
  26. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  27. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  28. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  29. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  30. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  31. 2023SpectralGPT: Spectral Remote Sensing Foundation Modelcited by 250
  32. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  33. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  34. 2019Image-based table recognition: data, model, and evaluationcited by 233
  35. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  36. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  37. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  38. 2024Pixtral 12Bcited by 174

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
  2. 2025LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?cited by 4
  3. 2026CodePercept: Code-Grounded Visual STEM Perception for MLLMscited by 3
  4. 2026StyleTextGen: Style-Conditioned Multilingual Scene Text Generationcited by 3
  5. 2025DocSLM: A Small Vision-Language Model for Long Multimodal Document Understandingcited by 2
  6. 2025MGA-VQA: Secure and Interpretable Graph-Augmented Visual Question Answering with Memory-Guided Protection Against Unauthorized Knowledge Usecited by 1
  7. 2026Q-Mask: Query-driven Causal Masks for Text Anchoring in OCR-Oriented Vision-Language Modelscited by 1
  8. 2026InstructTable: Improving Table Structure Recognition Through Instructionscited by 1
  9. 2026LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinementcited by 1
  10. 2026Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruningcited by 1
  11. 2026VaRS-Doc: Interpretation-Aware Variant Representations via Latent Self-Probing for Visual Document Retrievalcited by 0
  12. 2026AdaThinking-E: One-Token Entropy Regulation for Adaptive Thinkingcited by 0
1 more
  1. 2026Xiaomi-OCR-0 Technical Reportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
  2. 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
  3. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See
  4. 2026MonkeyOCRv2: A Visual-Text Foundation Model for Document AI
  5. 2026TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications
  6. 2023Efficient End-to-End Visual Document Understanding with Rationale Distillation
  7. 2025Index-Preserving Lightweight Token Pruning for Efficient Document Understanding in Vision-Language Models
  8. 2023Planting a SEED of Vision in Large Language Model