mlsanityData to 14 Oct 2026

HRVDA: High-Resolution Visual Document Assistant

arXiv 2404.06918 ↗First posted 10 Apr 2024cs.CVcited by 37

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Leveraging vast training data, multimodal large language models (MLLMs) have demonstrated formidable general visual comprehension capabilities and achieved remarkable performance across various tasks. However, their performance in visual document understanding still leaves much room for improvement. This discrepancy is primarily attributed to the fact that visual document understanding is a fine-grained prediction task. In natural scenes, MLLMs typically use low-resolution images, leading to a substantial loss of visual information. Furthermore, general-purpose MLLMs do not excel in handling document-oriented instructions. In this paper, we propose a High-Resolution Visual Document Assistant (HRVDA), which bridges the gap between MLLMs and visual document understanding. This model employs a content filtering mechanism and an instruction filtering module to separately filter out the content-agnostic visual tokens and instruction-agnostic visual tokens, thereby achieving efficient model training and inference for high-resolution images. In addition, we construct a document-oriented visual instruction tuning dataset and apply a multi-stage training strategy to enhance the model's document modeling capabilities. Extensive experiments demonstrate that our model achieves state-of-the-art performance across multiple document understanding datasets, while maintaining training efficiency and inference speed comparable to low-resolution models.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 67 of its 75 references that are in the map. First posted on arXiv 10 Apr 2024.

Builds on

67 of 75 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding4 Jul

Other references · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2020End-to-End Object Detection with Transformerscited by 7,414
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2019Towards VQA Models That Can Readcited by 1,718
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  10. 2022Token Merging: Your ViT But Fastercited by 1,021
  11. 2023A Survey on Multimodal Large Language Modelscited by 909
  12. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  13. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  14. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  15. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  16. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  17. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  18. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  19. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  20. 2021OCR-free Document Understanding Transformercited by 407
  21. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  22. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  23. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  24. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  25. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  26. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  27. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 256
  28. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 256
  29. 2019Image-based table recognition: data, model, and evaluationcited by 237
  30. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  31. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 213
  32. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 151
  33. 2019Shape Robust Text Detection with Progressive Scale Expansion Networkcited by 147
  34. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  35. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  36. 2021Learned Token Pruning for Transformerscited by 139
  37. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  38. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 15; 2026: 8.

37 papers.

Sort:

By area:

  1. 2025MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsingcited by 116
  2. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  3. 2024TextSquare: Scaling up Text-Centric Visual Instruction Tuningcited by 62
  4. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  5. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 38
  6. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  7. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  8. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  9. 2024Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramidcited by 26
  10. 2025WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?cited by 26
  11. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Explorationcited by 25
  12. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
25 more
  1. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  2. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  3. 2024Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Informationcited by 18
  4. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 14
  5. 2024Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reactioncited by 13
  6. 2024LLaVA-Read: Enhancing Reading Ability of Multimodal Language Modelscited by 12
  7. 2024StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyondcited by 11
  8. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 9
  9. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  10. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  11. 2024Deep Learning based Key Information Extraction from Business Documents: Systematic Literature Reviewcited by 6
  12. 2025Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Imagescited by 5
  13. 2026UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documentscited by 5
  14. 2026Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligencecited by 4
  15. 2024Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Modelscited by 3
  16. 2025Unsupervised Document and Template Clustering using Multimodal Embeddingscited by 3
  17. 2025M3DR: Towards Universal Multilingual Multimodal Document Retrievalcited by 3
  18. 2025DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Modelcited by 3
  19. 2026HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Frameworkcited by 3
  20. 2026MVP: Enhancing Video Large Language Models via Self-supervised Masked Video Predictioncited by 1
  21. 2026MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systemscited by 1
  22. 2026GVR-Coder: A Visual-Feedback Framework for Structured SVG Generation in Complex Document and Meeting Scenarioscited by 1
  23. 2025First-place Solution for Streetscape Shop Sign Recognition Competitioncited by 0
  24. 2026Kyrtos: A methodology for automatic deep analysis of graphic charts with curves in technical documentscited by 0
  25. 2026PositionOCR: Augmenting Positional Awareness in Multi-Modal Models via Hybrid Specialist Integrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
  2. 2024DocVLM: Make Your VLM an Efficient Reader
  3. 2024Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models
  4. 2026Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering
  5. 2026Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding
  6. 2024InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions
  7. 2024Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding
  8. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding