# Deep Learning for Video Classification and Captioning

arXiv: https://arxiv.org/abs/1609.06782 · HTML: https://mlsanity.com/paper/1609.06782/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2016.

## Built on by (most cited first, 28 of 28)

- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2303.13874/) (2023)
- [ST-LLM: Large Language Models Are Effective Temporal Learners](https://mlsanity.com/paper/2404.00308/) (2024)
- [Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?](https://mlsanity.com/paper/2301.00184/) (2022)
- [Disentangled Representation Learning for Text-Video Retrieval](https://mlsanity.com/paper/2203.07111/) (2022)
- [UATVR: Uncertainty-Adaptive Text-Video Retrieval](https://mlsanity.com/paper/2301.06309/) (2023)
- [Lost in Time: A New Temporal Benchmark for VideoLLMs](https://mlsanity.com/paper/2410.07752/) (2024)
- [Zero-Shot Video Captioning with Evolving Pseudo-Tokens](https://mlsanity.com/paper/2207.11100/) (2022)
- [PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance](https://mlsanity.com/paper/2411.02327/) (2024)
- [Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions with Multi-Level Representations](https://mlsanity.com/paper/2204.03382/) (2022)
- [From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping](https://mlsanity.com/paper/2304.13273/) (2023)
- [Boundary-Denoising for Video Activity Localization](https://mlsanity.com/paper/2304.02934/) (2023)
- [DynFocus: Dynamic Cooperative Network Empowers LLMs with Video Understanding](https://mlsanity.com/paper/2411.12355/) (2024)
- [VLM-Eval: A General Evaluation on Video Large Language Models](https://mlsanity.com/paper/2311.11865/) (2023)
- [Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach](https://mlsanity.com/paper/2408.07249/) (2024)
- [Intelligent Monitoring of Stress Induced by Water Deficiency in Plants using Deep Learning](https://mlsanity.com/paper/2104.07911/) (2021)
- [VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval](https://mlsanity.com/paper/2412.01558/) (2024)
- [DynImg: Key Frames with Visual Prompts are Good Representation for Multi-Modal Video Understanding](https://mlsanity.com/paper/2507.15569/) (2025)
- [A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2412.00049/) (2024)
- [A Deep Neural Framework for Image Caption Generation Using GRU-Based Attention Mechanism](https://mlsanity.com/paper/2203.01594/) (2022)
- [Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks](https://mlsanity.com/paper/2210.05038/) (2022)
- [Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understanding](https://mlsanity.com/paper/2505.12605/) (2025)
- [Enhancing Transformer for Video Understanding Using Gated Multi-Level Attention and Temporal Adversarial Training](https://mlsanity.com/paper/2103.10043/) (2021)
- [Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2202.13673/) (2022)
- [SnakeVoxFormer: Transformer-based Single ImageVoxel Reconstruction with Run Length Encoding](https://mlsanity.com/paper/2303.16293/) (2023)
- [A Comparative Study of Pre-trained CNNs and GRU-Based Attention for Image Caption Generation](https://mlsanity.com/paper/2310.07252/) (2023)
- [Images in Motion?: A First Look into Video Leakage in Collaborative Deep Learning](https://mlsanity.com/paper/2509.09742/) (2025)
- [Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedback](https://mlsanity.com/paper/2510.02561/) (2025)

## Abstract (arXiv)

Accelerated by the tremendous increase in Internet bandwidth and storage space, video data has been generated, published and spread explosively, becoming an indispensable part of today's big data. In this paper, we focus on reviewing two lines of research aiming to stimulate the comprehension of videos with deep learning: video classification and video captioning. While video classification concentrates on automatically labeling video clips based on their semantic contents like human actions or complex events, video captioning attempts to generate a complete and natural sentence, enriching the single label as in video classification, to capture the most informative dynamics in videos. In addition, we also provide a review of popular benchmarks and competitions, which are critical for evaluating the technical progress of this vibrant field.