mlsanityData to 14 Oct 2026

TAP: Text-Aware Pre-training for Text-VQA and Text-Caption

arXiv 2012.04638 ↗First posted 8 Dec 2020cs.CVcited by 95

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we propose Text-Aware Pre-training (TAP) for Text-VQA and Text-Caption tasks. These two tasks aim at reading and understanding scene text in images for question answering and image caption generation, respectively. In contrast to the conventional vision-language pre-training that fails to capture scene text and its relationship with the visual and text modalities, TAP explicitly incorporates scene text (generated from OCR engines) in pre-training. With three pre-training tasks, including masked language modeling (MLM), image-text (contrastive) matching (ITM), and relative (spatial) position prediction (RPP), TAP effectively helps the model learn a better aligned representation among the three modalities: text word, visual object, and scene text. Due to this aligned representation learning, even pre-trained on the same downstream task dataset, TAP already boosts the absolute accuracy on the TextVQA dataset by +5.4%, compared with a non-TAP baseline. To further improve the performance, we build a large-scale dataset based on the Conceptual Caption dataset, named OCR-CC, which contains 1.4 million scene text-related image-text pairs. Pre-trained on this OCR-CC dataset, our approach outperforms the state of the art by large margins on multiple tasks, i.e., +8.3% accuracy on TextVQA, +8.6% accuracy on ST-VQA, and +10.2 CIDEr score on TextCaps.

Key facts. Cited by 95 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 16; 2022: 29; 2023: 29; 2024: 12; 2025: 6; 2026: 3.

95 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  5. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  6. 2021InfographicVQAcited by 437
  7. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  8. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  9. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  10. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 245
  11. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  12. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
38 more
  1. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  2. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 150
  3. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  4. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  5. 2021Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Predictioncited by 131
  6. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  7. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  8. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  9. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  10. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  11. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  12. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  13. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  14. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 58
  15. 2022Vision-and-Language Pretrained Models: A Surveycited by 50
  16. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  17. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  18. 2022Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognitioncited by 45
  19. 2022TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformercited by 42
  20. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  21. 2023Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checkingcited by 35
  22. 2022MaskOCR: Text Recognition with Masked Encoder-Decoder Pretrainingcited by 34
  23. 2022Locality Guidance for Improving Vision Transformers on Tiny Datasetscited by 33
  24. 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMscited by 32
  25. 2022ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrievalcited by 29
  26. 2022PreSTU: Pre-Training for Scene-Text Understandingcited by 27
  27. 2022Self-supervised Character-to-Character Distillation for Text Recognitioncited by 27
  28. 2022Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answeringcited by 25
  29. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 25
  30. 2023Reading Between the Lanes: Text VideoQA on the Roadcited by 23
  31. 2023Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answeringcited by 21
  32. 2023Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literaturecited by 21
  33. 2022Watching the News: Towards VideoQA Models that can Readcited by 20
  34. 2025Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligencecited by 20
  35. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  36. 2023Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustnesscited by 19
  37. 2021PAM: Understanding Product Images in Cross Product Category Attribute Extractioncited by 18
  38. 2022Vision-Language Pre-Training for Boosting Scene Text Detectorscited by 18

50 most cited of 95 All 95 (JSON)