mlsanityData to 14 Oct 2026

Focus Anywhere for Fine-grained Multi-page Document Understanding

arXiv 2405.14295 ↗First posted 23 May 2024cs.CVcited by 62

Not on the map. Why?

Cited by 24 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Modern LVLMs still struggle to achieve fine-grained document understanding, such as OCR/translation/caption for regions of interest to the user, tasks that require the context of the entire page, or even multiple pages. Accordingly, this paper proposes Fox, an effective pipeline, hybrid data, and tuning strategy, that catalyzes LVLMs to focus anywhere on single/multi-page documents. We introduce a novel task to boost the document understanding by making LVLMs focus attention on the document-level region, such as redefining full-page OCR as foreground focus. We employ multiple vision vocabularies to extract visual hybrid knowledge for interleaved document pages (e.g., a page containing a photo). Meanwhile, we render cross-vocabulary vision data as the catalyzer to achieve a full reaction of multiple visual vocabularies and in-document figure understanding. Further, without modifying the weights of multiple vision vocabularies, the above catalyzed fine-grained understanding capabilities can be efficiently tuned to multi-page documents, enabling the model to focus anywhere in both format-free and page-free manners. Besides, we build a benchmark including 9 fine-grained sub-tasks (e.g., region-level OCR/summary, color-guided OCR) to promote document analysis in the community. The experimental results verify the superiority of our model.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

31 of 44 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Segment Anythingcited by 8,234
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2023Qwen Technical Reportcited by 3,344
19 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  9. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  10. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  11. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  12. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 245
  13. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 236
  14. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 174
  15. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  16. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  17. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  18. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  19. 2021PP-OCRv2: Bag of Tricks for Ultra Lightweight OCR Systemcited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 28; 2026: 20.

62 papers.

Sort:

By area:

  1. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 267
  2. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 183
  3. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 180
  4. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  5. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 150
  6. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  7. 2025Perception-R1: Pioneering Perception Policy with Reinforcement Learningcited by 120
  8. 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacycited by 81
  9. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 79
  10. 2026DeepSeek-OCR 2: Visual Causal Flowcited by 74
  11. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 36
  12. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
38 more
  1. 2025POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversioncited by 34
  2. 2026STEP3-VL-10B Technical Reportcited by 32
  3. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 26
  4. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
  5. 2024Slow Perception: Let's Perceive Geometric Figures Step-by-stepcited by 24
  6. 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Promptingcited by 22
  7. 2024Law of Vision Representation in MLLMscited by 18
  8. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 17
  9. 2025Context Cascade Compression: Exploring the Upper Limits of Text Compressioncited by 17
  10. 2024READoc: A Unified Benchmark for Realistic Document Structured Extractioncited by 15
  11. 2025UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Modelscited by 13
  12. 2025Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCRcited by 12
  13. 2025Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Modelscited by 12
  14. 2025DocThinker: Explainable Multimodal Large Language Models with Rule-based Reinforcement Learning for Document Understandingcited by 12
  15. 2026OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Modelscited by 12
  16. 2026DM0: An Embodied-Native Vision-Language-Action Model towards Physical AIcited by 12
  17. 2024DOGR: Towards Versatile Visual Document Grounding and Referringcited by 10
  18. 2025'Eclair -- Extracting Content and Layout with Integrated Reading Order for Documentscited by 9
  19. 2025PerPO: Perceptual Preference Optimization via Discriminative Rewardingcited by 9
  20. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 9
  21. 2025DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translationcited by 9
  22. 2025Reading or Reasoning? Format Decoupled Reinforcement Learning for Document OCRcited by 9
  23. 2026Visual Merit or Linguistic Crutch? A Close Look at DeepSeek-OCRcited by 8
  24. 2026MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decodingcited by 8
  25. 2024μgat: Improving Single-Page Document Parsing by Providing Multi-Page Contextcited by 7
  26. 2026Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Trainingcited by 7
  27. 2025Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?cited by 6
  28. 2026MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarioscited by 5
  29. 2024Arabic-Nougat: Fine-Tuning Vision Transformers for Arabic OCR and Markdown Extractioncited by 4
  30. 2025LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?cited by 4
  31. 2026Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligencecited by 4
  32. 2025GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decouplingcited by 3
  33. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 3
  34. 2025MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Modelscited by 3
  35. 2025MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUscited by 3
  36. 2025PILOT: A Promptable Interleaved Layout-aware OCR Transformercited by 2
  37. 2025PATIMT-Bench: A Multi-Scenario Benchmark for Position-Aware Text Image Machine Translation in Large Vision-Language Modelscited by 2
  38. 2026GutenOCR: A Grounded Vision-Language Front-End for Documentscited by 2

50 most cited of 62 All 62 (JSON)