# A Review of Deep Learning for Video Captioning

arXiv: https://arxiv.org/abs/2304.11431 · HTML: https://mlsanity.com/paper/2304.11431/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 213 of its 411 references that are in the map.
- First posted on arXiv 22 Apr 2023.

## Builds on (most cited first, 30 of 213)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)

## Built on by (most cited first, 28 of 28)

- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges](https://mlsanity.com/paper/2309.13925/) (2023)
- [Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives](https://mlsanity.com/paper/2406.05615/) (2024)
- [A Survey on Video Temporal Grounding with Multimodal Large Language Model](https://mlsanity.com/paper/2508.10922/) (2025)
- [Video Anomaly Detection in 10 Years: A Survey and Outlook](https://mlsanity.com/paper/2405.19387/) (2024)
- [Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocols](https://mlsanity.com/paper/2311.02538/) (2023)
- [When Thinking Drifts: Evidential Grounding for Robust Video Reasoning](https://mlsanity.com/paper/2510.06077/) (2025)
- [AURA: Always-On Understanding and Real-Time Assistance via Video Streams](https://mlsanity.com/paper/2604.04184/) (2026)
- [Enrich and Detect: Video Temporal Grounding with Multimodal LLMs](https://mlsanity.com/paper/2510.17023/) (2025)
- [Progress-Aware Video Frame Captioning](https://mlsanity.com/paper/2412.02071/) (2024)
- [CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning](https://mlsanity.com/paper/2606.09393/) (2026)
- [ChronusOmni: Improving Time Awareness of Omni Large Language Models](https://mlsanity.com/paper/2512.09841/) (2025)
- [NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularization](https://mlsanity.com/paper/2507.10894/) (2025)
- [ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2502.19409/) (2025)
- [Harnessing Streaming Video in the Wild](https://mlsanity.com/paper/2606.08615/) (2026)
- [UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark](https://mlsanity.com/paper/2410.01180/) (2024)
- [Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning](https://mlsanity.com/paper/2607.01667/) (2026)
- [SnapCap: Efficient Snapshot Compressive Video Captioning](https://mlsanity.com/paper/2401.04903/) (2024)
- [ViSIL: Unified Evaluation of Information Loss in Multimodal Video Captioning](https://mlsanity.com/paper/2601.09851/) (2026)
- [Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understanding](https://mlsanity.com/paper/2505.12605/) (2025)
- [From Captions to Keyframes: KeyScore for Multimodal Frame Scoring and Video-Language Understanding](https://mlsanity.com/paper/2510.06509/) (2025)
- [Process-of-Thought Reasoning for Videos](https://mlsanity.com/paper/2602.07689/) (2026)
- [VCEval: Rethinking What is a Good Educational Video and How to Automatically Evaluate It](https://mlsanity.com/paper/2407.12005/) (2024)
- [Multi-Modal interpretable automatic video captioning](https://mlsanity.com/paper/2411.06872/) (2024)
- [QCaption: Video Captioning and Q&A through Fusion of Large Multimodal Models](https://mlsanity.com/paper/2601.06566/) (2026)
- [MobiDiary: Autoregressive Action Captioning with Wearable Devices and Wireless Signals](https://mlsanity.com/paper/2601.08204/) (2026)
- [Training-Free Semantic Multi-Object Tracking with Vision-Language Models](https://mlsanity.com/paper/2604.14074/) (2026)
- [CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenes](https://mlsanity.com/paper/2608.18734/) (2026)

## Abstract (arXiv)

Video captioning (VC) is a fast-moving, cross-disciplinary area of research that bridges work in the fields of computer vision, natural language processing (NLP), linguistics, and human-computer interaction. In essence, VC involves understanding a video and describing it with language. Captioning is used in a host of applications from creating more accessible interfaces (e.g., low-vision navigation) to video question answering (V-QA), video retrieval and content generation. This survey covers deep learning-based VC, including but, not limited to, attention-based architectures, graph networks, reinforcement learning, adversarial networks, dense video captioning (DVC), and more. We discuss the datasets and evaluation metrics used in the field, and limitations, applications, challenges, and future directions for VC.