# Deep Reinforcement Learning-based Image Captioning with Embedding Reward

arXiv: https://arxiv.org/abs/1704.03899 · HTML: https://mlsanity.com/paper/1704.03899/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2017.

## Built on by (most cited first, 30 of 40)

- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Machine Translation Decoding beyond Beam Search](https://mlsanity.com/paper/2104.05336/) (2021)
- [Learning to Generate Better Than Your LLM](https://mlsanity.com/paper/2306.11816/) (2023)
- [Consensus Graph Representation Learning for Better Grounded Image Captioning](https://mlsanity.com/paper/2112.00974/) (2021)
- [Reinforcement Learning-powered Semantic Communication via Semantic Similarity](https://mlsanity.com/paper/2108.12121/) (2021)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [Prophet Attention: Predicting Attention with Future Attention for Image Captioning](https://mlsanity.com/paper/2210.10914/) (2022)
- [TCIC: Theme Concepts Learning Cross Language and Vision for Image Captioning](https://mlsanity.com/paper/2106.10936/) (2021)
- [CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning](https://mlsanity.com/paper/2602.21655/) (2026)
- [Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation](https://mlsanity.com/paper/2102.11387/) (2021)
- [Explicit Image Caption Editing](https://mlsanity.com/paper/2207.09625/) (2022)
- [Cross Modal Compression: Towards Human-comprehensible Semantic Compression](https://mlsanity.com/paper/2209.02574/) (2022)
- [Efficient Modeling of Future Context for Image Captioning](https://mlsanity.com/paper/2207.10897/) (2022)
- [Language Model Decoding as Likelihood-Utility Alignment](https://mlsanity.com/paper/2210.07228/) (2022)
- [From RAG to RICHES: Retrieval Interlaced with Sequence Generation](https://mlsanity.com/paper/2407.00361/) (2024)
- [PRPO: Paragraph-level Policy Optimization for Vision-Language Deepfake Detection](https://mlsanity.com/paper/2509.26272/) (2025)
- [Collaborative Multi-Agent Video Fast-Forwarding](https://mlsanity.com/paper/2305.17569/) (2023)
- [Disentangled Representation Learning for Controllable Person Image Generation](https://mlsanity.com/paper/2312.05798/) (2023)
- [A Survey on Large Language Models from Concept to Implementation](https://mlsanity.com/paper/2403.18969/) (2024)
- [Label-Attention Transformer with Geometrically Coherent Objects for Image Captioning](https://mlsanity.com/paper/2109.07799/) (2021)
- [Image Captioning using Multiple Transformers for Self-Attention Mechanism](https://mlsanity.com/paper/2103.05103/) (2021)
- [Which Discriminator for Cooperative Text Generation?](https://mlsanity.com/paper/2204.11586/) (2022)
- [Are metrics measuring what they should? An evaluation of image captioning task metrics](https://mlsanity.com/paper/2207.01733/) (2022)
- [Belief Revision based Caption Re-ranker with Visual Semantic Information](https://mlsanity.com/paper/2209.08163/) (2022)
- [Language-Informed Beam Search Decoding for Multilingual Machine Translation](https://mlsanity.com/paper/2408.05738/) (2024)
- [Transfer Reward Learning for Policy Gradient-Based Text Generation](https://mlsanity.com/paper/1909.03622/) (2019)
- [A Deep Neural Framework for Image Caption Generation Using GRU-Based Attention Mechanism](https://mlsanity.com/paper/2203.01594/) (2022)
- [ALCAP: Alignment-Augmented Music Captioner](https://mlsanity.com/paper/2212.10901/) (2022)
- [Agentic AI: The Era of Semantic Decoding](https://mlsanity.com/paper/2403.14562/) (2024)

## Abstract (arXiv)

Image captioning is a challenging problem owing to the complexity in understanding the image content and diverse ways of describing it in natural language. Recent advances in deep neural networks have substantially improved the performance of this task. Most state-of-the-art approaches follow an encoder-decoder framework, which generates captions using a sequential recurrent prediction model. However, in this paper, we introduce a novel decision-making framework for image captioning. We utilize a "policy network" and a "value network" to collaboratively generate captions. The policy network serves as a local guidance by providing the confidence of predicting the next word according to the current state. Additionally, the value network serves as a global and lookahead guidance by evaluating all possible extensions of the current state. In essence, it adjusts the goal of predicting the correct words towards the goal of generating captions similar to the ground truth captions. We train both networks using an actor-critic reinforcement learning model, with a novel reward defined by visual-semantic embedding. Extensive experiments and analyses on the Microsoft COCO dataset show that the proposed framework outperforms state-of-the-art approaches across different evaluation metrics.