# Improved Image Captioning via Policy Gradient optimization of SPIDEr

arXiv: https://arxiv.org/abs/1612.00370 · HTML: https://mlsanity.com/paper/1612.00370/

## Key facts

- Cited by 113 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2016.

## Built on by (most cited first, 30 of 113)

- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models](https://mlsanity.com/paper/2406.15704/) (2024)
- [X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning](https://mlsanity.com/paper/2311.18799/) (2023)
- [Audio-Visual LLM for Video Understanding](https://mlsanity.com/paper/2312.06720/) (2023)
- [Learning to Generate Better Than Your LLM](https://mlsanity.com/paper/2306.11816/) (2023)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.08826/) (2025)
- [LocCa: Visual Pretraining with Location-aware Captioners](https://mlsanity.com/paper/2403.19596/) (2024)
- [Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning](https://mlsanity.com/paper/2309.11500/) (2023)
- [EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning](https://mlsanity.com/paper/2401.17690/) (2024)
- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [Towards Accurate Text-based Image Captioning with Content Diversity Exploration](https://mlsanity.com/paper/2105.03236/) (2021)
- [Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation](https://mlsanity.com/paper/2410.10676/) (2024)
- [Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning](https://mlsanity.com/paper/2108.05067/) (2021)
- [BOLT: Fast Energy-based Controlled Text Generation with Tunable Biases](https://mlsanity.com/paper/2305.12018/) (2023)
- [MusCaps: Generating Captions for Music Audio](https://mlsanity.com/paper/2104.11984/) (2021)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [CIDEr-R: Robust Consensus-based Image Description Evaluation](https://mlsanity.com/paper/2109.13701/) (2021)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [Reinforced Video Captioning with Entailment Rewards](https://mlsanity.com/paper/1708.02300/) (2017)
- [Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks](https://mlsanity.com/paper/2204.07780/) (2022)
- [Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models](https://mlsanity.com/paper/2310.05863/) (2023)
- [SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis](https://mlsanity.com/paper/2106.01444/) (2021)
- [Leveraging Pre-trained BERT for Audio Captioning](https://mlsanity.com/paper/2203.02838/) (2022)
- [Improved Baselines for Data-efficient Perceptual Augmentation of LLMs](https://mlsanity.com/paper/2403.13499/) (2024)
- [Don't Take It Literally: An Edit-Invariant Sequence Loss for Text Generation](https://mlsanity.com/paper/2106.15078/) (2021)
- [Learning Spatially-Aware Language and Audio Embeddings](https://mlsanity.com/paper/2409.11369/) (2024)

## Abstract (arXiv)

Current image captioning methods are usually trained via (penalized) maximum likelihood estimation. However, the log-likelihood score of a caption does not correlate well with human assessments of quality. Standard syntactic evaluation metrics, such as BLEU, METEOR and ROUGE, are also not well correlated. The newer SPICE and CIDEr metrics are better correlated, but have traditionally been hard to optimize for. In this paper, we show how to use a policy gradient (PG) method to directly optimize a linear combination of SPICE and CIDEr (a combination we call SPIDEr): the SPICE score ensures our captions are semantically faithful to the image, while CIDEr score ensures our captions are syntactically fluent. The PG method we propose improves on the prior MIXER approach, by using Monte Carlo rollouts instead of mixing MLE training with PG. We show empirically that our algorithm leads to easier optimization and improved results compared to MIXER. Finally, we show that using our PG method we can optimize any of the metrics, including the proposed SPIDEr metric which results in image captions that are strongly preferred by human raters compared to captions generated by the same model but trained to optimize MLE or the COCO metrics.