Video Captioning: a comparative review of where we are and which could be the route
Not on the map. Why?
Abstract (arXiv)
Video captioning is the process of describing the content of a sequence of images capturing its semantic relationships and meanings. Dealing with this task with a single image is arduous, not to mention how difficult it is for a video (or images sequence). The amount and relevance of the applications of video captioning are vast, mainly to deal with a significant amount of video recordings in video surveillance, or assisting people visually impaired, to mention a few. To analyze where the efforts of our community to solve the video captioning task are, as well as what route could be better to follow, this manuscript presents an extensive review of more than 105 papers for the period of 2016 to 2021. As a result, the most-used datasets and metrics are identified. Also, the main approaches used and the best ones. We compute a set of rankings based on several performance metrics to obtain, according to its performance, the best method with the best result on the video captioning task. Finally, some insights are concluded about which could be the next steps or opportunity areas to improve dealing with this complex task.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 71 of its 185 references that are in the map. First posted on arXiv 12 Apr 2022.
Builds on
71 of 185 references.
References · 71
- 2017Attention Is All You Needcited by 59,932
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
- 2014Distributed Representations of Sentences and Documentscited by 677
- 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
- 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
- 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
- 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
- 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
- 2015A Dataset for Movie Descriptioncited by 195
38 more
- 2016Movie Descriptioncited by 180
- 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
- 2018ECO: Efficient Convolutional Network for Online Video Understandingcited by 115
- 2020Multi-modal Dense Video Captioningcited by 113
- 2020Spatio-Temporal Graph for Video Captioning with Knowledge Distillationcited by 109
- 2020Object Relational Graph with Teacher-Recommended Learning for Video Captioningcited by 106
- 2018Reconstruction Network for Video Captioningcited by 98
- 2020Multimodal Pretraining for Dense Video Captioningcited by 79
- 2017Attention-Based Multimodal Fusion for Video Descriptioncited by 75
- 2018Bidirectional Attentive Fusion with Context Gating for Dense Video Captioningcited by 68
- 2019Memory-Attended Recurrent Network for Video Captioningcited by 65
- 2016End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answeringcited by 60
- 2018Jointly Localizing and Describing Events for Dense Video Captioningcited by 60
- 2019Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioningcited by 60
- 2019Streamlined Dense Video Captioningcited by 57
- 2015Video Paragraph Captioning Using Hierarchical Recurrent Neural Networkscited by 54
- 2015Jointly Modeling Embedding and Translation to Bridge Video and Languagecited by 53
- 2018Less Is More: Picking Informative Frames for Video Captioningcited by 52
- 2016Semantic Compositional Networks for Visual Captioningcited by 50
- 2019Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Networkcited by 48
- 2017Video Captioning via Hierarchical Reinforcement Learningcited by 47
- 2016Video Captioning with Transferred Semantic Attributescited by 46
- 2014Coherent Multi-Sentence Video Description with Variable Level of Detailcited by 45
- 2019Object-aware Aggregation with Bidirectional Temporal Graph for Video Captioningcited by 42
- 2015Recognizing Fine-Grained and Composite Activities using Hand-Centric Features and Script Datacited by 40
- 2018Move Forward and Tell: A Progressive Generator of Video Descriptionscited by 35
- 2015Using Descriptive Video Services to Create a Large Data Source for Video Annotation Researchcited by 33
- 2021Semantic Grouping Network for Video Captioningcited by 33
- 2015Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioningcited by 30
- 2016Multimodal Memory Modelling for Video Captioningcited by 20
- 2016Hierarchical Boundary-Aware Neural Encoder for Video Captioningcited by 20
- 2020Learning to Discretely Compose Reasoning Module Networks for Video Captioningcited by 20
- 2017Hierarchical LSTM with Adjusted Temporal Attention for Video Captioningcited by 19
- 2017Weakly Supervised Dense Video Captioningcited by 18
- 2017Supervising Neural Attention Models for Video Captioning by Human Gaze Datacited by 18
- 2018Hierarchical LSTMs with Adaptive Attention for Visual Captioningcited by 18
- 2017Multi-Task Video Captioning with Video and Entailment Generationcited by 17
- 2021CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoisingcited by 17
50 most cited of 71 All 71 (JSON)
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.