mlsanityData to 7 Oct 2026

InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions

arXiv 2401.13313 ↗First posted 24 Jan 2024cs.CV cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

We study the problem of completing various visual document understanding (VDU) tasks, e.g., question answering and information extraction, on real-world documents through human-written instructions. To this end, we propose InstructDoc, the first large-scale collection of 30 publicly available VDU datasets, each with diverse instructions in a unified format, which covers a wide range of 12 tasks and includes open document types/formats. Furthermore, to enhance the generalization performance on VDU tasks, we design a new instruction-based document reading and understanding model, InstructDr, that connects document images, image encoders, and large language models (LLMs) through a trainable bridging module. Experiments demonstrate that InstructDr can effectively adapt to new VDU datasets, tasks, and domains via given instructions and outperforms existing multimodal LLMs and ChatGPT without specific training.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 24 Jan 2024.

Builds on

39 of 50 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2016A Diagram Is Worth A Dozen Imagescited by 793
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
27 more
  1. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  2. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  3. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  4. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  5. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  6. 2019Scene Text Visual Question Answeringcited by 327
  7. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  8. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  9. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  10. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  11. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  12. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  13. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  14. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  15. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  16. 2020DocBank: A Benchmark Dataset for Document Layout Analysiscited by 152
  17. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  18. 2022DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysiscited by 146
  19. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  20. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  21. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  22. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  23. 2021SciCap: Generating Captions for Scientific Figurescited by 76
  24. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  25. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  26. 2021Spatial Dual-Modality Graph Reasoning for Key Information Extractioncited by 43
  27. 2022Recognition-free Question Answering on Handwritten Document Collectionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 10; 2026: 5.

27 papers.

Sort:

By area:

  1. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
  2. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  3. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  4. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  5. 2024DocVLM: Make Your VLM an Efficient Readercited by 27
  6. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 26
  7. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  8. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  9. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  10. 2024Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understandingcited by 12
  11. 2024StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyondcited by 11
  12. 2024VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document Understandingcited by 8
15 more
  1. 2024DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthinesscited by 8
  2. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  3. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  4. 2024Notes on Applicability of GPT-4 to Document Understandingcited by 7
  5. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  6. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  7. 2026AVIR: Adaptive Visual In-Document Retrieval for Efficient Multi-Page Document Question Answeringcited by 4
  8. 2026Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligencecited by 4
  9. 2024CREPE: Coordinate-Aware End-to-End Document Parsercited by 3
  10. 2024DocKD: Knowledge Distillation from LLMs for Open-World Document Understanding Modelscited by 3
  11. 2025In-Context Learning can Perform Continual Learning Like Humanscited by 1
  12. 2026IDP Accelerator: Agentic Document Intelligence from Extraction to Compliance Validationcited by 1
  13. 2026Q-Mask: Query-driven Causal Masks for Text Anchoring in OCR-Oriented Vision-Language Modelscited by 1
  14. 2025DocPolarBERT: A Pre-trained Model for Document Understanding with Relative Polar Coordinate Encoding of Layout Structurescited by 0
  15. 2026SketchJudge: A Diagnostic Benchmark for Grading Hand-drawn Diagrams with Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering
  2. 2026Responses Fall Short of Understanding: Revealing the Gap between Internal Representations and Responses in Visual Document Understanding
  3. 2024HRVDA: High-Resolution Visual Document Assistant
  4. 2022Test-Time Adaptation for Visual Document Understanding
  5. 2025QID: Efficient Query-Informed ViTs in Data-Scarce Regimes for OCR-free Visual Document Understanding
  6. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
  7. 2025Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning
  8. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding