mlsanityData to 7 Oct 2026

PreSTU: Pre-Training for Scene-Text Understanding

arXiv 2209.05534 ↗First posted 12 Sep 2022cs.CV cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

The ability to recognize and reason about text embedded in visual inputs is often lacking in vision-and-language (V&L) models, perhaps because V&L pre-training methods have often failed to include such an ability in their training objective. In this paper, we propose PreSTU, a novel pre-training recipe dedicated to scene-text understanding (STU). PreSTU introduces OCR-aware pre-training objectives that encourage the model to recognize text from an image and connect it to the rest of the image content. We implement PreSTU using a simple transformer-based encoder-decoder architecture, combined with large-scale image-text datasets with scene text obtained from an off-the-shelf OCR system. We empirically demonstrate the effectiveness of this pre-training approach on eight visual question answering and four image captioning benchmarks.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 66 of its 86 references that are in the map. First posted on arXiv 12 Sep 2022.

Builds on

66 of 86 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
38 more
  1. 2019Towards VQA Models That Can Readcited by 1,696
  2. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2021Scaling Vision Transformerscited by 993
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  13. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  14. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  15. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  16. 2016A Diagram Is Worth A Dozen Imagescited by 793
  17. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  18. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  19. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  20. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  21. 2021InfographicVQAcited by 430
  22. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  23. 2021OCR-free Document Understanding Transformercited by 400
  24. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  25. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  26. 2019Scene Text Visual Question Answeringcited by 327
  27. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  28. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  29. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  30. 2016Incorporating Copying Mechanism in Sequence-to-Sequence Learningcited by 217
  31. 2014Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognitioncited by 214
  32. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  33. 2014Reading Text in the Wild with Convolutional Neural Networkscited by 122
  34. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  35. 2018FOTS: Fast Oriented Text Spotting with a Unified Networkcited by 109
  36. 2020Captioning Images Taken by People Who Are Blindcited by 108
  37. 2019Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQAcited by 107
  38. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 12; 2024: 7; 2025: 6.

27 papers.

Sort:

By area:

  1. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  2. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  3. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
  4. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  5. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  6. 2023ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rulescited by 24
  7. 2023Reading Between the Lanes: Text VideoQA on the Roadcited by 23
  8. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 22
  9. 2023Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustnesscited by 19
  10. 2025Visual Text Processing: A Comprehensive Review and Unified Evaluationcited by 18
  11. 2023Enhanced Chart Understanding in Vision and Language Task via Cross-modal Pre-training on Plot Table Pairscited by 15
  12. 2024Visually Guided Generative Text-Layout Pre-training for Document Intelligencecited by 12
15 more
  1. 2024Visual Text Meets Low-level Vision: A Comprehensive Survey on Visual Text Processingcited by 8
  2. 2024ViOCRVQA: Novel Benchmark Dataset and Vision Reader for Visual Question Answering by Understanding Vietnamese Text in Imagescited by 8
  3. 2023Separate and Locate: Rethink the Text in Text-based Visual Question Answeringcited by 7
  4. 2024ViTextVQA: A Large-Scale Visual Question Answering Dataset and a Novel Multimodal Feature Fusion Method for Vietnamese Text Comprehension in Imagescited by 7
  5. 2024InstructOCR: Instruction Boosting Scene Text Spottingcited by 7
  6. 2023Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Modelscited by 4
  7. 2023Multiple-Question Multiple-Answer Text-VQAcited by 4
  8. 2023SCOB: Universal Text Understanding via Character-wise Supervised Contrastive Learning with Online Text Rendering for Bridging Domain Gapcited by 3
  9. 2025Class-Agnostic Region-of-Interest Matching in Document Imagescited by 2
  10. 2025Restore Text First, Enhance Image Later: Two-Stage Scene Text Image Super-Resolution with Glyph Structure Guidancecited by 2
  11. 2025LiGT: Layout-infused Generative Transformer for Visual Question Answering on Vietnamese Receiptscited by 1
  12. 2023GenPlot: Increasing the Scale and Diversity of Chart Derendering Datacited by 0
  13. 2023A Surrogate-Assisted Extended Generative Adversarial Network for Parameter Optimization in Free-Form Metasurface Designcited by 0
  14. 2024ViConsFormer: Constituting Meaningful Phrases of Scene Texts using Transformer-based Method in Vietnamese Text-based Visual Question Answeringcited by 0
  15. 2025Towards Signboard-Oriented Visual Question Answering: ViSignVQA Dataset, Method and Benchmarkcited by 0