# Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning

arXiv: https://arxiv.org/abs/2402.13936 · HTML: https://mlsanity.com/paper/2402.13936/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Scaling Up Vision-Language Pre-training for Image Captioning](https://mlsanity.com/paper/2111.12233/) (2021)
- [Decoupled Contrastive Learning](https://mlsanity.com/paper/2110.06848/) (2021)
- [On the Weaknesses of Reinforcement Learning for Neural Machine Translation](https://mlsanity.com/paper/1907.01752/) (2019)
- [Fine-grained Image Captioning with CLIP Reward](https://mlsanity.com/paper/2205.13115/) (2022)
- [Discriminability objective for training descriptive captions](https://mlsanity.com/paper/1803.04376/) (2018)
- [Towards Diverse and Natural Image Descriptions via a Conditional GAN](https://mlsanity.com/paper/1703.06029/) (2017)
- [Contrastive Learning for Image Captioning](https://mlsanity.com/paper/1710.02534/) (2017)
- [Describing like humans: on diversity in image captioning](https://mlsanity.com/paper/1903.12020/) (2019)
- [Multimodal Knowledge Alignment with Reinforcement Learning](https://mlsanity.com/paper/2205.12630/) (2022)
- [Distinctive Image Captioning via CLIP Guided Group Optimization](https://mlsanity.com/paper/2208.04254/) (2022)
- [Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learning](https://mlsanity.com/paper/2212.03230/) (2022)

## Built on by (most cited first, 2 of 2)

- [Automatic Identification and Description of Jewelry Through Computer Vision and Neural Networks for Translators and Interpreters](https://mlsanity.com/paper/2509.00661/) (2025)
- [LineupRL: Verifiable Reinforcement Learning for Time Series Captioning via Caption-to-Series Identification](https://mlsanity.com/paper/2610.01800/) (2026)

## Abstract (arXiv)

Training image captioning models using teacher forcing results in very generic samples, whereas more distinctive captions can be very useful in retrieval applications or to produce alternative texts describing images for accessibility. Reinforcement Learning (RL) allows to use cross-modal retrieval similarity score between the generated caption and the input image as reward to guide the training, leading to more distinctive captions. Recent studies show that pre-trained cross-modal retrieval models can be used to provide this reward, completely eliminating the need for reference captions. However, we argue in this paper that Ground Truth (GT) captions can still be useful in this RL framework. We propose a new image captioning model training strategy that makes use of GT captions in different ways. Firstly, they can be used to train a simple MLP discriminator that serves as a regularization to prevent reward hacking and ensures the fluency of generated captions, resulting in a textual GAN setup extended for multimodal inputs. Secondly, they can serve as additional trajectories in the RL strategy, resulting in a teacher forcing loss weighted by the similarity of the GT to the image. This objective acts as an additional learning signal grounded to the distribution of the GT captions. Thirdly, they can serve as strong baselines when added to the pool of captions used to compute the proposed contrastive reward to reduce the variance of gradient estimate. Experiments on MS-COCO demonstrate the interest of the proposed training strategy to produce highly distinctive captions while maintaining high writing quality.