# Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures

arXiv: https://arxiv.org/abs/1601.03896 · HTML: https://mlsanity.com/paper/1601.03896/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jan 2016.

## Built on by (most cited first, 30 of 36)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Learning to Predict Visual Attributes in the Wild](https://mlsanity.com/paper/2106.09707/) (2021)
- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [INVIGORATE: Interactive Visual Grounding and Grasping in Clutter](https://mlsanity.com/paper/2108.11092/) (2021)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [Cross-Domain Image Captioning with Discriminative Finetuning](https://mlsanity.com/paper/2304.01662/) (2023)
- [Selective Replay Enhances Learning in Online Continual Analogical Reasoning](https://mlsanity.com/paper/2103.03987/) (2021)
- [MultiSubs: A Large-scale Multimodal and Multilingual Dataset](https://mlsanity.com/paper/2103.01910/) (2021)
- [Dealing with Semantic Underspecification in Multimodal NLP](https://mlsanity.com/paper/2306.05240/) (2023)
- [Keyword localisation in untranscribed speech using visually grounded speech models](https://mlsanity.com/paper/2202.01107/) (2022)
- [Diagnostic Captioning: A Survey](https://mlsanity.com/paper/2101.07299/) (2021)
- [The Role of Data Curation in Image Captioning](https://mlsanity.com/paper/2305.03610/) (2023)
- [Comparative evaluation of CNN architectures for Image Caption Generation](https://mlsanity.com/paper/2102.11506/) (2021)
- [Alt-Text with Context: Improving Accessibility for Images on Twitter](https://mlsanity.com/paper/2305.14779/) (2023)
- [ContextRef: Evaluating Referenceless Metrics For Image Description Generation](https://mlsanity.com/paper/2309.11710/) (2023)
- [Bangla Image Caption Generation through CNN-Transformer based Encoder-Decoder Network](https://mlsanity.com/paper/2110.12442/) (2021)
- [Are metrics measuring what they should? An evaluation of image captioning task metrics](https://mlsanity.com/paper/2207.01733/) (2022)
- [Vision-and-Language Pretraining](https://mlsanity.com/paper/2207.01772/) (2022)
- [Introducing Brain-like Concepts to Embodied Hand-crafted Dialog Management System](https://mlsanity.com/paper/2406.08996/) (2024)
- [Narration Generation for Cartoon Videos](https://mlsanity.com/paper/2101.06803/) (2021)
- [Visual representation of negation: Real world data analysis on comic image design](https://mlsanity.com/paper/2105.10131/) (2021)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [LecEval: An Automated Metric for Multimodal Knowledge Acquisition in Multimedia Learning](https://mlsanity.com/paper/2505.02078/) (2025)
- [Toward Reliable VLM: A Fine-Grained Benchmark and Framework for Exposure, Bias, and Inference in Korean Street Views](https://mlsanity.com/paper/2506.03371/) (2025)
- [Studying Illustrations in Manuscripts: An Efficient Deep-Learning Approach](https://mlsanity.com/paper/2601.05269/) (2025)
- [Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models](https://mlsanity.com/paper/2608.06901/) (2026)
- [Characterization and recognition of handwritten digits using Julia](https://mlsanity.com/paper/2102.11994/) (2021)
- [We went to look for meaning and all we got were these lousy representations: aspects of meaning representation for computational semantics](https://mlsanity.com/paper/2109.04949/) (2021)
- [Zero-shot Cross-Linguistic Learning of Event Semantics](https://mlsanity.com/paper/2207.02356/) (2022)

## Abstract (arXiv)

Automatic description generation from natural images is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities. In this survey, we classify the existing approaches based on how they conceptualize this problem, viz., models that cast description as either generation problem or as a retrieval problem over a visual or multimodal representational space. We provide a detailed review of existing models, highlighting their advantages and disadvantages. Moreover, we give an overview of the benchmark image datasets and the evaluation measures that have been developed to assess the quality of machine-generated image descriptions. Finally we extrapolate future directions in the area of automatic image description generation.