mlsanityData to 7 Oct 2026

VisualMRC: Machine Reading Comprehension on Document Images

arXiv 2101.11272 ↗First posted 27 Jan 2021cs.CL cs.CVcited by 150

Not on the map. Why?

Cited by 64 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent studies on machine reading comprehension have focused on text-level understanding but have not yet reached the level of human understanding of the visual layout and content of real-world documents. In this study, we introduce a new visual machine reading comprehension dataset, named VisualMRC, wherein given a question and a document image, a machine reads and comprehends texts in the image to answer the question in natural language. Compared with existing visual question answering (VQA) datasets that contain texts in images, VisualMRC focuses more on developing natural language understanding and generation abilities. It contains 30,000+ pairs of a question and an abstractive answer for 10,000+ document images sourced from multiple domains of webpages. We also introduce a new model that extends existing sequence-to-sequence models, pre-trained with large-scale text corpora, to take into account the visual layout and content of documents. Experiments with VisualMRC show that this model outperformed the base sequence-to-sequence models and a state-of-the-art VQA model. However, its performance is still below that of humans on most automatic evaluation metrics. The dataset will facilitate research aimed at connecting vision and language understanding.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 27 Jan 2021.

Builds on

39 of 51 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2016Layer Normalizationcited by 5,579
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
27 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  9. 2018Self-Attention with Relative Position Representationscited by 1,112
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  12. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  13. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  14. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  15. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  16. 2019Scene Text Visual Question Answeringcited by 327
  17. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  18. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  19. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  20. 2019Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQAcited by 107
  21. 2018Chargrid: Towards Understanding 2D Documentscited by 100
  22. 2017Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networkcited by 92
  23. 2020On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answeringcited by 78
  24. 2018RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipescited by 74
  25. 2020RikiNet: Reading Wikipedia Pages for Natural Question Answeringcited by 21
  26. 2020Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialogcited by 20
  27. 2020Video-Grounded Dialogues with Pretrained Generation Language Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 6; 2023: 24; 2024: 53; 2025: 42; 2026: 20.

150 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  4. 2021InfographicVQAcited by 430
  5. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  6. 2024What matters when building vision-language models?cited by 321
  7. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  8. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  9. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  10. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  11. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  12. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
38 more
  1. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  2. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  3. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  4. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  5. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  6. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  7. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  8. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  9. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  10. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  11. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  12. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  13. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  14. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  15. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  16. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  17. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94
  18. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
  19. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  20. 2021Document AI: Benchmarks, Models and Applicationscited by 61
  21. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  22. 2024A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understandingcited by 57
  23. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
  24. 2025FineVision: Open Data Is All You Needcited by 51
  25. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  26. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  27. 2024UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysiscited by 40
  28. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
  29. 2025MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resourcescited by 37
  30. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  31. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  32. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  33. 2023PDFVQA: A New Dataset for Real-World VQA on PDF Documentscited by 28
  34. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  35. 2024Harnessing Webpage UIs for Text-Rich Visual Understandingcited by 28
  36. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  37. 2024InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructionscited by 27
  38. 2024TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Modelscited by 27

50 most cited of 150 All 150 (JSON)