# Reinforced Video Captioning with Entailment Rewards

arXiv: https://arxiv.org/abs/1708.02300 · HTML: https://mlsanity.com/paper/1708.02300/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 7 Aug 2017.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [A Decomposable Attention Model for Natural Language Inference](https://mlsanity.com/paper/1606.01933/) (2016)
- [Recurrent Neural Network Regularization](https://mlsanity.com/paper/1409.2329/) (2014)
- [Long Short-Term Memory-Networks for Machine Reading](https://mlsanity.com/paper/1601.06733/) (2016)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [Improved Image Captioning via Policy Gradient optimization of SPIDEr](https://mlsanity.com/paper/1612.00370/) (2016)
- [Translating Videos to Natural Language Using Deep Recurrent Neural Networks](https://mlsanity.com/paper/1412.4729/) (2014)
- [Describing Videos by Exploiting Temporal Structure](https://mlsanity.com/paper/1502.08029/) (2015)
- [Reasoning about Entailment with Neural Attention](https://mlsanity.com/paper/1509.06664/) (2015)
- [Jointly Modeling Embedding and Translation to Bridge Video and Language](https://mlsanity.com/paper/1505.01861/) (2015)
- [Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning](https://mlsanity.com/paper/1511.03476/) (2015)
- [Multi-Task Video Captioning with Video and Entailment Generation](https://mlsanity.com/paper/1704.07489/) (2017)
- [Reinforcement Learning Neural Turing Machines - Revised](https://mlsanity.com/paper/1505.00521/) (2015)
- [Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text](https://mlsanity.com/paper/1604.01729/) (2016)

## Built on by (most cited first, 21 of 21)

- [RL with KL penalties is better viewed as Bayesian inference](https://mlsanity.com/paper/2205.11275/) (2022)
- [Aligning Language Models with Preferences through f-divergence Minimization](https://mlsanity.com/paper/2302.08215/) (2023)
- [On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting](https://mlsanity.com/paper/2206.00761/) (2022)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Relational Graph Learning for Grounded Video Description Generation](https://mlsanity.com/paper/2112.00967/) (2021)
- [ENTRUST: Argument Reframing with Language Models and Entailment](https://mlsanity.com/paper/2103.06758/) (2021)
- [Energy-Based Models for Code Generation under Compilability Constraints](https://mlsanity.com/paper/2106.04985/) (2021)
- [Controllable Video Captioning with an Exemplar Sentence](https://mlsanity.com/paper/2112.01073/) (2021)
- [The MSR-Video to Text Dataset with Clean Annotations](https://mlsanity.com/paper/2102.06448/) (2021)
- [IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning](https://mlsanity.com/paper/2507.18531/) (2025)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [Automatic Document Sketching: Generating Drafts from Analogous Texts](https://mlsanity.com/paper/2106.07192/) (2021)
- [Nested-Wasserstein Self-Imitation Learning for Sequence Generation](https://mlsanity.com/paper/2001.06944/) (2020)
- [Variational Stacked Local Attention Networks for Diverse Video Captioning](https://mlsanity.com/paper/2201.00985/) (2022)
- [DANTE-AD: Dual-Vision Attention Network for Long-Term Audio Description](https://mlsanity.com/paper/2503.24096/) (2025)
- [Syntax Customized Video Captioning by Imitating Exemplar Sentences](https://mlsanity.com/paper/2112.01062/) (2021)
- [Reward Models are Metrics in a Trench Coat](https://mlsanity.com/paper/2510.03231/) (2025)
- [The Role of the Input in Natural Language Video Description](https://mlsanity.com/paper/2102.05067/) (2021)
- [Visual-aware Attention Dual-stream Decoder for Video Captioning](https://mlsanity.com/paper/2110.08578/) (2021)
- [METEOR Guided Divergence for Video Captioning](https://mlsanity.com/paper/2212.10690/) (2022)
- [Diverse Sign Language Translation](https://mlsanity.com/paper/2410.19586/) (2024)

## Abstract (arXiv)

Sequence-to-sequence models have shown promising improvements on the temporal task of video captioning, but they optimize word-level cross-entropy loss during training. First, using policy gradient and mixed-loss methods for reinforcement learning, we directly optimize sentence-level task-based metrics (as rewards), achieving significant improvements over the baseline, based on both automatic metrics and human evaluation on multiple datasets. Next, we propose a novel entailment-enhanced reward (CIDEnt) that corrects phrase-matching based metrics (such as CIDEr) to only allow for logically-implied partial matches and avoid contradictions, achieving further significant improvements over the CIDEr-reward model. Overall, our CIDEnt-reward model achieves the new state-of-the-art on the MSR-VTT dataset.