mlsanityData to 7 Oct 2026

DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding

arXiv 2311.11810 ↗First posted 20 Nov 2023cs.CV cs.AIcited by 106

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

This work presents DocPedia, a novel large multimodal model (LMM) for versatile OCR-free document understanding, capable of parsing images up to 2,560×2,560 resolution. Unlike existing work either struggle with high-resolution documents or give up the large language model thus vision or language ability constrained, our DocPedia directly processes visual input in the frequency domain rather than the pixel space. The unique characteristic enables DocPedia to capture a greater amount of visual and textual information using a limited number of visual tokens. To consistently enhance both perception and comprehension abilities of our model, we develop a dual-stage training strategy and enrich instructions/annotations of all training tasks covering multiple document types. Extensive quantitative and qualitative experiments conducted on various publicly available benchmarks confirm the mutual benefits of jointly learning perception and comprehension tasks. The results provide further evidence of the effectiveness and superior performance of our DocPedia over other methods.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 65 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

51 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding4 Jul

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  7. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  9. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  10. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  11. 2021InfographicVQAcited by 430
  12. 2021OCR-free Document Understanding Transformercited by 400
  13. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  14. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  15. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  16. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  17. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  18. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  19. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  20. 2019Real-time Scene Text Detection with Differentiable Binarizationcited by 212
  21. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  22. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  23. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  24. 2018FOTS: Fast Oriented Text Spotting with a Unified Networkcited by 109
  25. 2021BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentscited by 104
  26. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  27. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  28. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  29. 2021StrucTexT: Structured Text Understanding with Multi-Modal Transformerscited by 90
  30. 2023UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understandingcited by 74
  31. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  32. 2023Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solutioncited by 54
  33. 2019ICDAR 2019 Competition on Scene Text Visual Question Answeringcited by 46
  34. 2023GeoLayoutLM: Geometric Pre-training for Visual Information Extractioncited by 40
  35. 2023StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingcited by 39
  36. 2022The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Trainingcited by 33
  37. 2022Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understandingcited by 20
  38. 2021NomMer: Nominate Synergistic Context in Vision Transformer for Visual Recognitioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 37; 2025: 32; 2026: 36.

106 papers.

Sort:

By area:

  1. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  2. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  3. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  4. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  5. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  6. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  7. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  8. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  9. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 78
  10. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 70
  11. 2024Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidancecited by 63
  12. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
38 more
  1. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  2. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  3. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  4. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 45
  5. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  6. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  7. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  8. 2025Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoningcited by 34
  9. 2024Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramidcited by 26
  10. 2025WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?cited by 26
  11. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  12. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understandingcited by 22
  13. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 22
  14. 2024The Evolution of Multimodal Model Architecturescited by 21
  15. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  16. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  17. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  18. 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Promptingcited by 20
  19. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  20. 2024HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Modelscited by 16
  21. 2024READoc: A Unified Benchmark for Realistic Document Structured Extractioncited by 15
  22. 2024Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Networkcited by 15
  23. 2024PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Samplingcited by 14
  24. 2025Taylor Series-Inspired Local Structure Fitting Network for Few-shot Point Cloud Semantic Segmentationcited by 14
  25. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  26. 2023UPOCR: Towards Unified Pixel-Level OCR Interfacecited by 12
  27. 2024LLaVA-Read: Enhancing Reading Ability of Multimodal Language Modelscited by 12
  28. 2024StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyondcited by 11
  29. 2024FlexAttention for Efficient High-Resolution Vision-Language Modelscited by 9
  30. 2025OCSU: Optical Chemical Structure Understanding for Molecule-centric Scientific Discoverycited by 8
  31. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  32. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  33. 2024Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Schedulingcited by 7
  34. 2024Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrievalcited by 6
  35. 2024Deep Learning based Key Information Extraction from Business Documents: Systematic Literature Reviewcited by 6
  36. 2025Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmarkcited by 5
  37. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  38. 2025Task-Oriented 6-DoF Grasp Pose Detection in Clutterscited by 4

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding
  2. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
  3. 2025Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark
  4. 2025PP-DocBee: Improving Multimodal Document Understanding Through a Bag of Tricks
  5. 2023DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents
  6. 2024HRVDA: High-Resolution Visual Document Assistant
  7. 2024Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding
  8. 2025Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning