mlsanityData to 7 Oct 2026

A Review of Deep Learning for Video Captioning

arXiv 2304.11431 ↗First posted 22 Apr 2023cs.CVcited by 28

Not on the map. Why?

Abstract (arXiv)

Video captioning (VC) is a fast-moving, cross-disciplinary area of research that bridges work in the fields of computer vision, natural language processing (NLP), linguistics, and human-computer interaction. In essence, VC involves understanding a video and describing it with language. Captioning is used in a host of applications from creating more accessible interfaces (e.g., low-vision navigation) to video question answering (V-QA), video retrieval and content generation. This survey covers deep learning-based VC, including but, not limited to, attention-based architectures, graph networks, reinforcement learning, adversarial networks, dense video captioning (DVC), and more. We discuss the datasets and evaluation metrics used in the field, and limitations, applications, challenges, and future directions for VC.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 213 of its 411 references that are in the map. First posted on arXiv 22 Apr 2023.

Builds on

213 of 411 references.

References · 213

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2018How Powerful are Graph Neural Networks?cited by 4,355
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  10. 2016Improved Techniques for Training GANscited by 2,938
  11. 2018YOLOv3: An Incremental Improvementcited by 2,652
  12. 2014Conditional Generative Adversarial Netscited by 2,152
38 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2020Knowledge Distillation: A Surveycited by 1,578
  5. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  6. 2019Simplifying Graph Convolutional Networkscited by 1,470
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  9. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  11. 2021Transformers in Vision: A Surveycited by 1,009
  12. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  13. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  14. 2017Dense-Captioning Events in Videoscited by 789
  15. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  16. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  17. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  18. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  19. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  20. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  21. 2020A Survey of Deep Active Learningcited by 547
  22. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  23. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  24. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  25. 2017Non-Autoregressive Neural Machine Translationcited by 457
  26. 2021Intriguing Properties of Vision Transformerscited by 446
  27. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  28. 2021A Survey of Transformerscited by 399
  29. 2018Deep Learning on Graphs: A Surveycited by 393
  30. 2019Video Instance Segmentationcited by 391
  31. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  32. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 346
  33. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  34. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  35. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  36. 2016Image Captioning with Semantic Attentioncited by 260
  37. 2017Automated Curriculum Learning for Neural Networkscited by 251
  38. 2021I-BERT: Integer-only BERT Quantizationcited by 250

50 most cited of 213 All 213 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 8; 2026: 10.

28 papers.

Sort:

By area:

  1. 2023Video Understanding with Large Language Models: A Surveycited by 203
  2. 2023Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challengescited by 42
  3. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  4. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  5. 2024Video Anomaly Detection in 10 Years: A Survey and Outlookcited by 18
  6. 2023Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocolscited by 17
  7. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 15
  8. 2026AURA: Always-On Understanding and Real-Time Assistance via Video Streamscited by 15
  9. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  10. 2024Progress-Aware Video Frame Captioningcited by 10
  11. 2025NavComposer: Composing Language Instructions for Navigation Trajectories through Action-Scene-Object Modularizationcited by 5
  12. 2025ChronusOmni: Improving Time Awareness of Omni Large Language Modelscited by 5
16 more
  1. 2025ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Modelscited by 4
  2. 2024UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmarkcited by 3
  3. 2026Harnessing Streaming Video in the Wildcited by 3
  4. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 3
  5. 2024SnapCap: Efficient Snapshot Compressive Video Captioningcited by 2
  6. 2026ViSIL: Unified Evaluation of Information Loss in Multimodal Video Captioningcited by 2
  7. 2025Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understandingcited by 1
  8. 2025From Captions to Keyframes: KeyScore for Multimodal Frame Scoring and Video-Language Understandingcited by 1
  9. 2026Process-of-Thought Reasoning for Videoscited by 1
  10. 2026Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioningcited by 1
  11. 2024VCEval: Rethinking What is a Good Educational Video and How to Automatically Evaluate Itcited by 0
  12. 2024Multi-Modal interpretable automatic video captioningcited by 0
  13. 2026QCaption: Video Captioning and Q&A through Fusion of Large Multimodal Modelscited by 0
  14. 2026MobiDiary: Autoregressive Action Captioning with Wearable Devices and Wireless Signalscited by 0
  15. 2026Training-Free Semantic Multi-Object Tracking with Vision-Language Modelscited by 0
  16. 2026CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenescited by 0