# Actor-Critic Sequence Training for Image Captioning

arXiv: https://arxiv.org/abs/1706.09601 · HTML: https://mlsanity.com/paper/1706.09601/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2017.

## Built on by (most cited first, 13 of 13)

- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Consensus Graph Representation Learning for Better Grounded Image Captioning](https://mlsanity.com/paper/2112.00974/) (2021)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [Label-Attention Transformer with Geometrically Coherent Objects for Image Captioning](https://mlsanity.com/paper/2109.07799/) (2021)
- [An Integral Projection-based Semantic Autoencoder for Zero-Shot Learning](https://mlsanity.com/paper/2306.14628/) (2023)
- [Image Captioning using Multiple Transformers for Self-Attention Mechanism](https://mlsanity.com/paper/2103.05103/) (2021)
- [Are metrics measuring what they should? An evaluation of image captioning task metrics](https://mlsanity.com/paper/2207.01733/) (2022)
- [Transfer Reward Learning for Policy Gradient-Based Text Generation](https://mlsanity.com/paper/1909.03622/) (2019)
- [Goal-driven text descriptions for images](https://mlsanity.com/paper/2108.12575/) (2021)
- [Cross Modification Attention Based Deliberation Model for Image Captioning](https://mlsanity.com/paper/2109.08411/) (2021)
- [Bridging the Gap Between Training and Inference of Bayesian Controllable Language Models](https://mlsanity.com/paper/2206.05519/) (2022)
- [MGRL-RSCC: Multi-Granularity Reward Reinforcement Learning for Fine-Grained Remote Sensing Change Captioning](https://mlsanity.com/paper/2609.26166/) (2026)

## Abstract (arXiv)

Generating natural language descriptions of images is an important capability for a robot or other visual-intelligence driven AI agent that may need to communicate with human users about what it is seeing. Such image captioning methods are typically trained by maximising the likelihood of ground-truth annotated caption given the image. While simple and easy to implement, this approach does not directly maximise the language quality metrics we care about such as CIDEr. In this paper we investigate training image captioning methods based on actor-critic reinforcement learning in order to directly optimise non-differentiable quality metrics of interest. By formulating a per-token advantage and value computation strategy in this novel reinforcement learning based captioning model, we show that it is possible to achieve the state of the art performance on the widely used MSCOCO benchmark.