mlsanityData to 14 Oct 2026

From Show to Tell: A Survey on Deep Learning-based Image Captioning

arXiv 2107.06912 ↗First posted 14 Jul 2021cs.CV cs.CLcited by 138

Not on the map. Why?

Cited by 24 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Connecting Vision and Language plays an essential role in Generative Intelligence. For this reason, large research efforts have been devoted to image captioning, i.e. describing images with syntactically and semantically meaningful sentences. Starting from 2015 the task has generally been addressed with pipelines composed of a visual encoder and a language model for text generation. During these years, both components have evolved considerably through the exploitation of object regions, attributes, the introduction of multi-modal connections, fully-attentive approaches, and BERT-like early-fusion strategies. However, regardless of the impressive results, research in image captioning has not reached a conclusive answer yet. This work aims at providing a comprehensive overview of image captioning approaches, from visual encoding and text generation to training strategies, datasets, and evaluation metrics. In this respect, we quantitatively compare many relevant state-of-the-art approaches to identify the most impactful technical innovations in architectures and training strategies. Moreover, many variants of the problem and its open challenges are discussed. The final goal of this work is to serve as a tool for understanding the existing literature and highlighting the future directions for a research area where Computer Vision and Natural Language Processing can find an optimal synergy.

Key facts. Cited by 138 papers in the map (data through 14 Oct 2026). Builds on 175 of its 257 references that are in the map. First posted on arXiv 14 Jul 2021.

Builds on

175 of 257 references.

References · 175

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  8. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2014Going Deeper with Convolutionscited by 4,241
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  11. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  15. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  16. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  17. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  18. 2016Generative Adversarial Text to Image Synthesiscited by 735
  19. 2018Object Hallucination in Image Captioningcited by 669
  20. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  21. 2016Self-critical Sequence Training for Image Captioningcited by 547
  22. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  23. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  24. 2018nocaps: novel object captioning at scalecited by 444
  25. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  26. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  27. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  28. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  29. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  30. 2019Attention on Attention for Image Captioningcited by 289
  31. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  32. 2016Image Captioning with Semantic Attentioncited by 260
  33. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 255
  34. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  35. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232
  36. 2019Connecting Vision and Language with Localized Narrativescited by 226
  37. 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
  38. 2017On the Automatic Generation of Medical Imaging Reportscited by 216

50 most cited of 175 All 175 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 15; 2023: 44; 2024: 34; 2025: 28; 2026: 15.

138 papers.

Sort:

By area:

  1. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  2. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
  3. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  4. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 185
  5. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  6. 2023GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Textcited by 82
  7. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
  8. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuningcited by 69
  9. 2023Text + Sketch: Image Compression at Ultra Low Ratescited by 67
  10. 2023Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Reviewcited by 62
  11. 2023PLIP: Language-Image Pre-training for Person Representation Learningcited by 49
  12. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
38 more
  1. 2023Changes to Captions: An Attentive Network for Remote Sensing Change Captioningcited by 41
  2. 2023BAGM: A Backdoor Attack for Manipulating Text-to-Image Generative Modelscited by 37
  3. 2022A Review of Emerging Research Directions in Abstract Visual Reasoningcited by 33
  4. 2022SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Textcited by 30
  5. 2023Segment and Caption Anythingcited by 30
  6. 2023A Survey on the Robustness of Computer Vision Models against Common Corruptionscited by 21
  7. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  8. 2022GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinementcited by 20
  9. 2022Affection: Learning Affective Explanations for Real-World Visual Datacited by 19
  10. 2023Mitigating Open-Vocabulary Caption Hallucinationscited by 19
  11. 2023With a Little Help from your own Past: Prototypical Memory Networks for Image Captioningcited by 18
  12. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectivescited by 18
  13. 2022The LAM Dataset: A Novel Benchmark for Line-Level Handwritten Text Recognitioncited by 17
  14. 2024Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancementcited by 16
  15. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  16. 2025Top-Down Semantic Refinement for Image Captioningcited by 16
  17. 2023Cross-Modal Causal Intervention for Medical Report Generationcited by 15
  18. 2023MMNet: Multi-Collaboration and Multi-Supervision Network for Sequential Deepfake Detectioncited by 15
  19. 2022CaMEL: Mean Teacher Learning for Image Captioningcited by 14
  20. 2023Cross-Domain Image Captioning with Discriminative Finetuningcited by 14
  21. 2023Improving Cross-modal Alignment with Synthetic Pairs for Text-only Image Captioningcited by 14
  22. 2024Improving Large Models with Small models: Lower Costs and Better Performancecited by 14
  23. 2024A Lightweight Sparse Focus Transformer for Remote Sensing Image Change Captioningcited by 13
  24. 2022A Frustratingly Simple Approach for End-to-End Image Captioningcited by 12
  25. 2023Diffusion Based Augmentation for Captioning and Retrieval in Cultural Heritagecited by 12
  26. 2024VIXEN: Visual Text Comparison Network for Image Difference Captioningcited by 12
  27. 2024ReMOVE: A Reference-free Metric for Object Erasurecited by 12
  28. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  29. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  30. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  31. 2023HAAV: Hierarchical Aggregation of Augmented Views for Image Captioningcited by 10
  32. 2024FodFoM: Fake Outlier Data by Foundation Models Creates Stronger Visual Out-of-Distribution Detectorcited by 10
  33. 2024Enhancing Descriptive Captions with Visual Attributes for Multimodal Perceptioncited by 10
  34. 2023Negative Pre-aware for Noisy Cross-modal Matchingcited by 9
  35. 2024Customized large language models can outperform Community Notes in correcting misinformationcited by 9
  36. 2023Sample-Efficient Learning of Novel Visual Conceptscited by 8
  37. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 8
  38. 2021Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasetscited by 7

50 most cited of 138 All 138 (JSON)