mlsanityData to 13 Oct 2026

Towards Models that Can See and Read

arXiv 2301.07389 ↗First posted 18 Jan 2023cs.CV cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Visual Question Answering (VQA) and Image Captioning (CAP), which are among the most popular vision-language tasks, have analogous scene-text versions that require reasoning from the text in the image. Despite their obvious resemblance, the two are treated independently and, as we show, yield task-specific methods that can either see or read, but not both. In this work, we conduct an in-depth analysis of this phenomenon and propose UniTNT, a Unified Text-Non-Text approach, which grants existing multimodal architectures scene-text understanding capabilities. Specifically, we treat scene-text information as an additional modality, fusing it with any pretrained encoder-decoder-based architecture via designated modules. Thorough experiments reveal that UniTNT leads to the first single model that successfully handles both task types. Moreover, we show that scene-text understanding capabilities can boost vision-language models' performance on general VQA and CAP by up to 2.69% and 0.6 CIDEr, respectively.

Key facts. Cited by 15 papers in the map (data through 13 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 18 Jan 2023.

Builds on

52 of 67 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  9. 2015VQA: Visual Question Answeringcited by 2,495
  10. 2019Towards VQA Models That Can Readcited by 1,705
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,616
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,452
38 more
  1. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,267
  3. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  4. 2021Florence: A New Foundation Model for Computer Visioncited by 803
  5. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 764
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 744
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 659
  8. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  9. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 573
  10. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 551
  11. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  12. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  13. 2018nocaps: novel object captioning at scalecited by 444
  14. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  15. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  16. 2019Scene Text Visual Question Answeringcited by 327
  17. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  18. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  19. 2019Attention on Attention for Image Captioningcited by 289
  20. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  21. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  22. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  23. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  24. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  25. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 186
  26. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110
  27. 2019Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQAcited by 107
  28. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  29. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  30. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  31. 2020SCATTER: Selective Context Attentional Scene Text Recognizercited by 69
  32. 2020MiniVLM: A Smaller and Faster Vision-Language Modelcited by 51
  33. 2021UFO: A UniFied TransfOrmer for Vision-Language Representation Learningcited by 46
  34. 2020Spatially Aware Multimodal Transformers for TextVQAcited by 44
  35. 2022Towards Weakly-Supervised Text Spotting using a Multi-Task Transformercited by 41
  36. 2021Towards Accurate Text-based Image Captioning with Content Diversity Explorationcited by 25
  37. 2020Structured Multimodal Attentions for TextVQAcited by 23
  38. 2020Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answeringcited by 19

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 3; 2024: 9; 2025: 2; 2026: 1.

15 papers.

Sort:

By area:

  1. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmarkcited by 55
  2. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them Firstcited by 39
  3. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captionscited by 33
  4. 2024Question Aware Vision Transformer for Multimodal Reasoningcited by 32
  5. 2024DocVLM: Make Your VLM an Efficient Readercited by 28
  6. 2024GRAM: Global Reasoning for Multi-Page VQAcited by 23
  7. 2023Parrot Captions Teach CLIP to Spot Textcited by 16
  8. 2023CLIPAG: Towards Generator-Free Text-to-Image Generationcited by 12
  9. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 12
  10. 2024VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document Understandingcited by 8
  11. 2024Text-to-Image Generation Via Energy-Based CLIPcited by 3
  12. 2025DocSLM: A Small Vision-Language Model for Long Multimodal Document Understandingcited by 2
3 more
  1. 2026DODO: Discrete OCR Diffusion Modelscited by 2
  2. 2024TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Modelscited by 1
  3. 2025BioD2C: A Dual-level Semantic Consistency Constraint Framework for Biomedical VQAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2026ReVA: A Region-Aware Visual Assistant for Visually Grounded Question Answering
  4. 2026Inference-Time Structural Reasoning for Compositional Vision-Language Understanding
  5. 2025SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In Text-only LLMs
  6. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts
  7. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
  8. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions