mlsanityData to 7 Oct 2026

TextCaps: a Dataset for Image Captioning with Reading Comprehension

arXiv 2003.12462 ↗First posted 24 Mar 2020cs.CV cs.CLcited by 398

Not on the map. Why?

Cited by 213 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Image descriptions can help visually impaired people to quickly understand the image content. While we made significant progress in automatically describing images and optical character recognition, current approaches are unable to include written text in their descriptions, although text is omnipresent in human environments and frequently critical to understand our surroundings. To study how to comprehend text in the context of an image we collect a novel dataset, TextCaps, with 145k captions for 28k images. Our dataset challenges a model to recognize text, relate it to its visual context, and decide what part of the text to copy or paraphrase, requiring spatial, semantic, and visual reasoning between multiple text tokens and visual entities, such as objects. We study baselines and adapt existing approaches to this new task, which we refer to as image captioning with reading comprehension. Our analysis with automatic and human studies shows that our new TextCaps dataset provides many new technical challenges over previous datasets.

Key facts. Cited by 398 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 19; 2023: 70; 2024: 145; 2025: 107; 2026: 39.

398 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  11. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
38 more
  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  3. 2024Yi: Open Foundation Models by 01.AIcited by 706
  4. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  7. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  8. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  9. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  10. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  11. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  12. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  13. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  14. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  15. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  16. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  17. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  18. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  19. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  20. 2024What matters when building vision-language models?cited by 321
  21. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  22. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  23. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  24. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  25. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  26. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  27. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  28. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  29. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 240
  30. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  31. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  32. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  33. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  34. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  35. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  36. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  37. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  38. 2024Building and better understanding vision-language models: insights and future directionscited by 169

50 most cited of 398 All 398 (JSON)