# Delving Deeper into the Decoder for Video Captioning

arXiv: https://arxiv.org/abs/2001.05614 · HTML: https://mlsanity.com/paper/2001.05614/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2020.

## Built on by (most cited first, 13 of 13)

- [End-to-end Generative Pretraining for Multimodal Video Captioning](https://mlsanity.com/paper/2201.08264/) (2022)
- [CLIP4Caption: CLIP for Video Caption](https://mlsanity.com/paper/2110.06615/) (2021)
- [DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training](https://mlsanity.com/paper/2303.03032/) (2023)
- [Learning Audio-Video Modalities from Image Captions](https://mlsanity.com/paper/2204.00679/) (2022)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Zero-Shot Video Captioning with Evolving Pseudo-Tokens](https://mlsanity.com/paper/2207.11100/) (2022)
- [Question-Answering Dense Video Events](https://mlsanity.com/paper/2409.04388/) (2024)
- [The MSR-Video to Text Dataset with Clean Annotations](https://mlsanity.com/paper/2102.06448/) (2021)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training](https://mlsanity.com/paper/2302.09736/) (2023)
- [RETTA: Retrieval-Enhanced Test-Time Adaptation for Zero-Shot Video Captioning](https://mlsanity.com/paper/2405.07046/) (2024)
- [MSVD-Indonesian: A Benchmark for Multimodal Video-Text Tasks in Indonesian](https://mlsanity.com/paper/2306.11341/) (2023)
- [Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning](https://mlsanity.com/paper/2608.11013/) (2026)

## Abstract (arXiv)

Video captioning is an advanced multi-modal task which aims to describe a video clip using a natural language sentence. The encoder-decoder framework is the most popular paradigm for this task in recent years. However, there exist some problems in the decoder of a video captioning model. We make a thorough investigation into the decoder and adopt three techniques to improve the performance of the model. First of all, a combination of variational dropout and layer normalization is embedded into a recurrent unit to alleviate the problem of overfitting. Secondly, a new online method is proposed to evaluate the performance of a model on a validation set so as to select the best checkpoint for testing. Finally, a new training strategy called professional learning is proposed which uses the strengths of a captioning model and bypasses its weaknesses. It is demonstrated in the experiments on Microsoft Research Video Description Corpus (MSVD) and MSR-Video to Text (MSR-VTT) datasets that our model has achieved the best results evaluated by BLEU, CIDEr, METEOR and ROUGE-L metrics with significant gains of up to 18% on MSVD and 3.5% on MSR-VTT compared with the previous state-of-the-art models.