# Deep Learning Approaches on Image Captioning: A Review

arXiv: https://arxiv.org/abs/2201.12944 · HTML: https://mlsanity.com/paper/2201.12944/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 101 of its 173 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 101)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)

## Built on by (most cited first, 30 of 43)

- [OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization](https://mlsanity.com/paper/2312.04403/) (2023)
- [A Review of Multi-Modal Large Language and Vision Models](https://mlsanity.com/paper/2404.01322/) (2024)
- [AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?](https://mlsanity.com/paper/2410.21259/) (2024)
- [A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes](https://mlsanity.com/paper/2403.07469/) (2024)
- [Question-Instructed Visual Descriptions for Zero-Shot Video Question Answering](https://mlsanity.com/paper/2402.10698/) (2024)
- [RankCLIP: Ranking-Consistent Language-Image Pretraining](https://mlsanity.com/paper/2404.09387/) (2024)
- [EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanations](https://mlsanity.com/paper/2506.24016/) (2025)
- [Transforming Dental Diagnostics with Artificial Intelligence: Advanced Integration of ChatGPT and Large Language Models for Patient Care](https://mlsanity.com/paper/2406.06616/) (2024)
- [DescribeEarth: Describe Anything for Remote Sensing Images](https://mlsanity.com/paper/2509.25654/) (2025)
- [VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions](https://mlsanity.com/paper/2509.25818/) (2025)
- [A Survey on Image-text Multimodal Models](https://mlsanity.com/paper/2309.15857/) (2023)
- [Beyond Single Frames: Can LMMs Comprehend Temporal and Contextual Narratives in Image Sequences?](https://mlsanity.com/paper/2502.13925/) (2025)
- [RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction](https://mlsanity.com/paper/2505.22613/) (2025)
- [An Ensemble Model with Attention Based Mechanism for Image Captioning](https://mlsanity.com/paper/2501.14828/) (2025)
- [GraphT5: Unified Molecular Graph-Language Modeling via Multi-Modal Cross-Token Attention](https://mlsanity.com/paper/2503.07655/) (2025)
- [When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation](https://mlsanity.com/paper/2601.04609/) (2026)
- [Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models](https://mlsanity.com/paper/2607.10640/) (2026)
- [Beyond Images: An Integrative Multi-modal Approach to Chest X-Ray Report Generation](https://mlsanity.com/paper/2311.11090/) (2023)
- [Cognitive resilience: Unraveling the proficiency of image-captioning models to interpret masked visual content](https://mlsanity.com/paper/2403.15876/) (2024)
- [Inserting Faces inside Captions: Image Captioning with Attention Guided Merging](https://mlsanity.com/paper/2405.02305/) (2024)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Pixels to Prose: Understanding the art of Image Captioning](https://mlsanity.com/paper/2408.15714/) (2024)
- [Effectively Enhancing Vision Language Large Models by Prompt Augmentation and Caption Utilization](https://mlsanity.com/paper/2409.14484/) (2024)
- [Perception of Visual Content: Differences Between Humans and Foundation Models](https://mlsanity.com/paper/2411.18968/) (2024)
- [Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation](https://mlsanity.com/paper/2506.05399/) (2025)
- [Emergent Natural Language with Communication Games for Improving Image Captioning Capabilities without Additional Data](https://mlsanity.com/paper/2507.08610/) (2025)
- [Siamese-Driven Optimization for Low-Resolution Image Latent Embedding in Image Captioning](https://mlsanity.com/paper/2512.08873/) (2025)
- [From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme](https://mlsanity.com/paper/2512.24555/) (2025)
- [Compressed Image Captioning using CNN-based Encoder-Decoder Framework](https://mlsanity.com/paper/2404.18062/) (2024)
- [Evaluating authenticity and quality of image captions via sentiment and semantic analyses](https://mlsanity.com/paper/2409.09560/) (2024)

## Abstract (arXiv)

Image captioning is a research area of immense importance, aiming to generate natural language descriptions for visual content in the form of still images. The advent of deep learning and more recently vision-language pre-training techniques has revolutionized the field, leading to more sophisticated methods and improved performance. In this survey paper, we provide a structured review of deep learning methods in image captioning by presenting a comprehensive taxonomy and discussing each method category in detail. Additionally, we examine the datasets commonly employed in image captioning research, as well as the evaluation metrics used to assess the performance of different captioning models. We address the challenges faced in this field by emphasizing issues such as object hallucination, missing context, illumination conditions, contextual understanding, and referring expressions. We rank different deep learning methods' performance according to widely used evaluation metrics, giving insight into the current state of the art. Furthermore, we identify several potential future directions for research in this area, which include tackling the information misalignment problem between image and text modalities, mitigating dataset bias, incorporating vision-language pre-training methods to enhance caption generation, and developing improved evaluation tools to accurately measure the quality of image captions.