mlsanityData to 7 Oct 2026

Deep Learning Approaches on Image Captioning: A Review

arXiv 2201.12944 ↗First posted 31 Jan 2022cs.CVcited by 43

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Image captioning is a research area of immense importance, aiming to generate natural language descriptions for visual content in the form of still images. The advent of deep learning and more recently vision-language pre-training techniques has revolutionized the field, leading to more sophisticated methods and improved performance. In this survey paper, we provide a structured review of deep learning methods in image captioning by presenting a comprehensive taxonomy and discussing each method category in detail. Additionally, we examine the datasets commonly employed in image captioning research, as well as the evaluation metrics used to assess the performance of different captioning models. We address the challenges faced in this field by emphasizing issues such as object hallucination, missing context, illumination conditions, contextual understanding, and referring expressions. We rank different deep learning methods' performance according to widely used evaluation metrics, giving insight into the current state of the art. Furthermore, we identify several potential future directions for research in this area, which include tackling the information misalignment problem between image and text modalities, mitigating dataset bias, incorporating vision-language pre-training methods to enhance caption generation, and developing improved evaluation tools to accurately measure the quality of image captions.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 101 of its 173 references that are in the map. First posted on arXiv 31 Jan 2022.

Builds on

101 of 173 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov

Other references · 100

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2016Densely Connected Convolutional Networkscited by 6,691
38 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  6. 2015Fast R-CNNcited by 2,930
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Matching Networks for One Shot Learningcited by 2,246
  9. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  10. 2014Conditional Generative Adversarial Netscited by 2,152
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  13. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  14. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  15. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  16. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  17. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  18. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  19. 2020Pre-Trained Image Processing Transformercited by 891
  20. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  21. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  22. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  23. 2018Object Hallucination in Image Captioningcited by 661
  24. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  25. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  26. 2017Convolutional Sequence to Sequence Learningcited by 550
  27. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  28. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  29. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  30. 2018nocaps: novel object captioning at scalecited by 444
  31. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  32. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  33. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  34. 2015LSTM: A Search Space Odysseycited by 315
  35. 2017Relation Networks for Object Detectioncited by 305
  36. 2018Graph R-CNN for Scene Graph Generationcited by 296
  37. 2019Attention on Attention for Image Captioningcited by 289
  38. 2015Stacked Attention Networks for Image Question Answeringcited by 273

50 most cited of 100 All 100 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 14; 2025: 20; 2026: 6.

43 papers.

Sort:

By area:

  1. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimizationcited by 36
  2. 2024A Review of Multi-Modal Large Language and Vision Modelscited by 30
  3. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 19
  4. 2024A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenescited by 10
  5. 2024RankCLIP: Ranking-Consistent Language-Image Pretrainingcited by 9
  6. 2024Question-Instructed Visual Descriptions for Zero-Shot Video Question Answeringcited by 8
  7. 2025EXPERT: An Explainable Image Captioning Evaluation Metric with Structured Explanationscited by 6
  8. 2024Transforming Dental Diagnostics with Artificial Intelligence: Advanced Integration of ChatGPT and Large Language Models for Patient Carecited by 5
  9. 2025DescribeEarth: Describe Anything for Remote Sensing Imagescited by 5
  10. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captionscited by 5
  11. 2023A Survey on Image-text Multimodal Modelscited by 4
  12. 2025Beyond Single Frames: Can LMMs Comprehend Temporal and Contextual Narratives in Image Sequences?cited by 4
31 more
  1. 2025RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstructioncited by 4
  2. 2025An Ensemble Model with Attention Based Mechanism for Image Captioningcited by 2
  3. 2026When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluationcited by 2
  4. 2026Spectral Heat Flow for Conservative Token Condensation in Vision-Language Modelscited by 2
  5. 2023Beyond Images: An Integrative Multi-modal Approach to Chest X-Ray Report Generationcited by 1
  6. 2024Cognitive resilience: Unraveling the proficiency of image-captioning models to interpret masked visual contentcited by 1
  7. 2024Inserting Faces inside Captions: Image Captioning with Attention Guided Mergingcited by 1
  8. 2024Pixels to Prose: Understanding the art of Image Captioningcited by 1
  9. 2024Effectively Enhancing Vision Language Large Models by Prompt Augmentation and Caption Utilizationcited by 1
  10. 2024Perception of Visual Content: Differences Between Humans and Foundation Modelscited by 1
  11. 2025GraphT5: Unified Molecular Graph-Language Modeling via Multi-Modal Cross-Token Attentioncited by 1
  12. 2025Attention-based transformer models for image captioning across languages: An in-depth survey and evaluationcited by 1
  13. 2025Emergent Natural Language with Communication Games for Improving Image Captioning Capabilities without Additional Datacited by 1
  14. 2025Siamese-Driven Optimization for Low-Resolution Image Latent Embedding in Image Captioningcited by 1
  15. 2025From Perception to Punchline: Empowering VLM with the Art of In-the-wild Memecited by 1
  16. 2024Compressed Image Captioning using CNN-based Encoder-Decoder Frameworkcited by 0
  17. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 0
  18. 2024Evaluating authenticity and quality of image captions via sentiment and semantic analysescited by 0
  19. 2025HalCECE: A Framework for Explainable Hallucination Detection through Conceptual Counterfactuals in Image Captioningcited by 0
  20. 2025Tri-FusionNet: Enhancing Image Description Generation with Transformer-based Fusion Network and Dual Attention Mechanismcited by 0
  21. 2025Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigmscited by 0
  22. 2025Da Yu: Towards USV-Based Image Captioning for Waterway Surveillance and Scene Understandingcited by 0
  23. 2025RORPCap: Retrieval-based Objects and Relations Prompt for Image Captioningcited by 0
  24. 2025Automatic Identification and Description of Jewelry Through Computer Vision and Neural Networks for Translators and Interpreterscited by 0
  25. 2025Pre-Trained CNN Architecture for Transformer-Based Image Caption Generation Modelcited by 0
  26. 2025SilhouetteTell: Practical Video Identification Leveraging Blurred Recordings of Video Subtitlescited by 0
  27. 2025From Pixels to Posts: Retrieval-Augmented Fashion Captioning and Hashtag Generationcited by 0
  28. 2026Standardizing Longitudinal Radiology Report Evaluation via Large Language Model Annotationcited by 0
  29. 2026Brazilian Portuguese Image Captioning with Transformers: A Study on Cross-Native-Translated Datasetcited by 0
  30. 2026MUNIChus: Multilingual News Image Captioning Benchmarkcited by 0
  31. 2026ECOKV: Geometry-Aware KV Cache Eviction via Complementary Diversity Metricscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking and Improving Detail Image Caption
  2. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
  3. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning
  4. 2024DreamLIP: Language-Image Pre-training with Long Captions
  5. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  6. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning
  7. 2026BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model
  8. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions