mlsanityData to 7 Oct 2026

LaTr: Layout-Aware Transformer for Scene-Text VQA

arXiv 2112.12494 ↗First posted 23 Dec 2021cs.CVcited by 72

Not on the map. Why?

Abstract (arXiv)

We propose a novel multimodal architecture for Scene Text Visual Question Answering (STVQA), named Layout-Aware Transformer (LaTr). The task of STVQA requires models to reason over different modalities. Thus, we first investigate the impact of each modality, and reveal the importance of the language module, especially when enriched with layout information. Accounting for this, we propose a single objective pre-training scheme that requires only text and spatial cues. We show that applying this pre-training scheme on scanned documents has certain advantages over using natural images, despite the domain gap. Scanned documents are easy to procure, text-dense and have a variety of layouts, helping the model learn various spatial cues (e.g. left-of, below etc.) by tying together language and layout information. Compared to existing approaches, our method performs vocabulary-free decoding and, as shown, generalizes well beyond the training vocabulary. We further demonstrate that LaTr improves robustness towards OCR errors, a common reason for failure cases in STVQA. In addition, by leveraging a vision transformer, we eliminate the need for an external object detector. LaTr outperforms state-of-the-art STVQA methods on multiple datasets. In particular, +7.6% on TextVQA, +10.8% on ST-VQA and +4.0% on OCR-VQA (all absolute accuracy numbers).

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 56 of its 78 references that are in the map. First posted on arXiv 23 Dec 2021.

Builds on

56 of 78 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
38 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2015Pointer Networkscited by 882
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  13. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  14. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  15. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  16. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  17. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  18. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  19. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  20. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  21. 2019Scene Text Visual Question Answeringcited by 327
  22. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  23. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  24. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  25. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  26. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  27. 2021Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognitioncited by 194
  28. 2019What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysiscited by 191
  29. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  30. 2019Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQAcited by 107
  31. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  32. 2020On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answeringcited by 78
  33. 2019Fusion of Detected Objects in Text for Visual Question Answeringcited by 74
  34. 2020Sequence-to-Sequence Contrastive Learning for Text Recognitioncited by 70
  35. 2020SCATTER: Selective Context Attentional Scene Text Recognizercited by 69
  36. 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
  37. 2019ICDAR 2019 Competition on Scene Text Visual Question Answeringcited by 46
  38. 2020Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Textcited by 46

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 30; 2024: 18; 2025: 9; 2026: 1.

72 papers.

Sort:

By area:

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  2. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  3. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  5. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  6. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  7. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  8. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  9. 2022Hierarchical multimodal transformers for Multi-Page DocVQAcited by 102
  10. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  11. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  12. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
38 more
  1. 2024Reconstructive Visual Instruction Tuningcited by 54
  2. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  3. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  4. 2022OCR-IDL: OCR Annotations for Industry Document Library Datasetcited by 35
  5. 2022VRDU: A Benchmark for Visually-rich Document Understandingcited by 35
  6. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 31
  7. 2022DocEnTr: An End-to-End Document Image Enhancement Transformercited by 28
  8. 2022PreSTU: Pre-Training for Scene-Text Understandingcited by 27
  9. 2024DocVLM: Make Your VLM an Efficient Readercited by 27
  10. 2022Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answeringcited by 24
  11. 2023ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rulescited by 24
  12. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  13. 2023Reading Between the Lanes: Text VideoQA on the Roadcited by 23
  14. 2023Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answeringcited by 21
  15. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  16. 2022Watching the News: Towards VideoQA Models that can Readcited by 20
  17. 2023Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literaturecited by 20
  18. 2022DocSegTr: An Instance-Level End-to-End Document Image Segmentation Transformercited by 19
  19. 2023DocTr: Document Transformer for Structured Information Extraction in Documentscited by 18
  20. 2022YORO -- Lightweight End to End Visual Groundingcited by 16
  21. 2023CLIPTER: Looking at the Bigger Picture in Scene Text Recognitioncited by 16
  22. 2023Generate then Select: Open-ended Visual Question Answering Guided by World Knowledgecited by 16
  23. 2024Enhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusioncited by 16
  24. 2023Towards Models that Can See and Readcited by 15
  25. 2022TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generationcited by 12
  26. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 12
  27. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  28. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learningcited by 10
  29. 2023Fully Authentic Visual Question Answering Dataset from Online Communitiescited by 9
  30. 2024ViOCRVQA: Novel Benchmark Dataset and Vision Reader for Visual Question Answering by Understanding Vietnamese Text in Imagescited by 8
  31. 2023Separate and Locate: Rethink the Text in Text-based Visual Question Answeringcited by 7
  32. 2023Analyzing the Efficacy of an LLM-Only Approach for Image-based Document Question Answeringcited by 7
  33. 2023DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Modelscited by 7
  34. 2024ViTextVQA: A Large-Scale Visual Question Answering Dataset and a Novel Multimodal Feature Fusion Method for Vietnamese Text Comprehension in Imagescited by 7
  35. 2024μgat: Improving Single-Page Document Parsing by Providing Multi-Page Contextcited by 7
  36. 2023Benchmarking Diverse-Modal Entity Linking with Generative Modelscited by 5
  37. 2022MUST-VQA: MUltilingual Scene-text VQAcited by 4
  38. 2023Locate Then Generate: Bridging Vision and Language with Bounding Box for Scene-Text VQAcited by 4

50 most cited of 72 All 72 (JSON)