# Language Models for Image Captioning: The Quirks and What Works

arXiv: https://arxiv.org/abs/1505.01809 · HTML: https://mlsanity.com/paper/1505.01809/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2015.

## Built on by (most cited first, 19 of 19)

- [Linearly Mapping from Image to Text Space](https://mlsanity.com/paper/2209.15162/) (2022)
- [TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model](https://mlsanity.com/paper/2503.18278/) (2025)
- [Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores](https://mlsanity.com/paper/2403.00553/) (2024)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Arithmetic Sampling: Parallel Diverse Decoding for Large Language Models](https://mlsanity.com/paper/2210.15458/) (2022)
- [Guiding Image Captioning Models Toward More Specific Captions](https://mlsanity.com/paper/2307.16686/) (2023)
- [DynRsl-VLM: Enhancing Autonomous Driving Perception with Dynamic Resolution Vision-Language Models](https://mlsanity.com/paper/2503.11265/) (2025)
- [Multi-Modal Image Captioning for the Visually Impaired](https://mlsanity.com/paper/2105.08106/) (2021)
- [From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment](https://mlsanity.com/paper/2406.13912/) (2024)
- [Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Data](https://mlsanity.com/paper/2305.15218/) (2023)
- [Rethinking Nearest Neighbors for Visual Classification](https://mlsanity.com/paper/2112.08459/) (2021)
- [Nearest Neighbors Meet Deep Neural Networks for Point Cloud Analysis](https://mlsanity.com/paper/2303.00703/) (2023)
- [Text-Only Training for Visual Storytelling](https://mlsanity.com/paper/2308.08881/) (2023)
- [Geometry-Entangled Visual Semantic Transformer for Image Captioning](https://mlsanity.com/paper/2109.14137/) (2021)
- [Multi-Glimpse Network: A Robust and Efficient Classification Architecture based on Recurrent Downsampled Attention](https://mlsanity.com/paper/2111.02018/) (2021)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [It Isn't Sh!tposting, It's My CAT Posting](https://mlsanity.com/paper/2205.08710/) (2022)
- [Targeted Image Data Augmentation Increases Basic Skills Captioning Robustness](https://mlsanity.com/paper/2309.15991/) (2023)
- [Compressed Image Captioning using CNN-based Encoder-Decoder Framework](https://mlsanity.com/paper/2404.18062/) (2024)

## Abstract (arXiv)

Two recent approaches have achieved state-of-the-art results in image captioning. The first uses a pipelined process where a set of candidate words is generated by a convolutional neural network (CNN) trained on images, and then a maximum entropy (ME) language model is used to arrange these words into a coherent sentence. The second uses the penultimate activation layer of the CNN as input to a recurrent neural network (RNN) that then generates the caption sequence. In this paper, we compare the merits of these different language modeling approaches for the first time by using the same state-of-the-art CNN as input. We examine issues in the different approaches, including linguistic irregularities, caption repetition, and data set overlap. By combining key aspects of the ME and RNN methods, we achieve a new record performance over previously published results on the benchmark COCO dataset. However, the gains we see in BLEU do not translate to human judgments.