# Efficient Video Classification Using Fewer Frames

arXiv: https://arxiv.org/abs/1902.10640 · HTML: https://mlsanity.com/paper/1902.10640/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2019.

## Built on by (most cited first, 26 of 26)

- [MoViNets: Mobile Video Networks for Efficient Video Recognition](https://mlsanity.com/paper/2103.11511/) (2021)
- [SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events](https://mlsanity.com/paper/2103.15538/) (2021)
- [Efficient Video Transformers with Spatial-Temporal Token Selection](https://mlsanity.com/paper/2111.11591/) (2021)
- [How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting](https://mlsanity.com/paper/2203.04781/) (2022)
- [A Cognitive-Based Trajectory Prediction Approach for Autonomous Driving](https://mlsanity.com/paper/2402.19251/) (2024)
- [DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval](https://mlsanity.com/paper/2106.13266/) (2021)
- [Gate-Shift-Fuse for Video Action Recognition](https://mlsanity.com/paper/2203.08897/) (2022)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [AZTR: Aerial Video Action Recognition with Auto Zoom and Temporal Reasoning](https://mlsanity.com/paper/2303.01589/) (2023)
- [SeiT: Storage-Efficient Vision Training with Tokens Using 1% of Pixel Storage](https://mlsanity.com/paper/2303.11114/) (2023)
- [Exploring Navigation Maps for Learning-Based Motion Prediction](https://mlsanity.com/paper/2302.06195/) (2023)
- [FrameHopper: Selective Processing of Video Frames in Detection-driven Real-Time Video Analytics](https://mlsanity.com/paper/2203.11493/) (2022)
- [A Closer Look at Spatial-Slice Features Learning for COVID-19 Detection](https://mlsanity.com/paper/2404.01643/) (2024)
- [Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review](https://mlsanity.com/paper/2207.00782/) (2022)
- [Video Content Classification using Deep Learning](https://mlsanity.com/paper/2111.13813/) (2021)
- [View-Invariant Skeleton-based Action Recognition via Global-Local Contrastive Learning](https://mlsanity.com/paper/2209.11634/) (2022)
- [StreamTinyNet: video streaming analysis with spatial-temporal TinyML](https://mlsanity.com/paper/2407.17524/) (2024)
- [TempNet: Temporal Attention Towards the Detection of Animal Behaviour in Videos](https://mlsanity.com/paper/2211.09950/) (2022)
- [Knowledge Distillation for Efficient Audio-Visual Video Captioning](https://mlsanity.com/paper/2306.09947/) (2023)
- [Judging a video by its bitstream cover](https://mlsanity.com/paper/2309.07361/) (2023)
- [HaltingVT: Adaptive Token Halting Transformer for Efficient Video Recognition](https://mlsanity.com/paper/2401.04975/) (2024)
- [Leveraging Compressed Frame Sizes For Ultra-Fast Video Classification](https://mlsanity.com/paper/2403.08580/) (2024)
- [Enhancing Video Understanding: Deep Neural Networks for Spatiotemporal Analysis](https://mlsanity.com/paper/2502.07277/) (2025)
- [Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixup](https://mlsanity.com/paper/2503.02284/) (2025)
- [Communicative Agents for Slideshow Storytelling Video Generation based on LLMs](https://mlsanity.com/paper/2509.01277/) (2025)
- [Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs](https://mlsanity.com/paper/2609.10355/) (2026)

## Abstract (arXiv)

Recently,there has been a lot of interest in building compact models for video classification which have a small memory footprint (<1 GB). While these models are compact, they typically operate by repeated application of a small weight matrix to all the frames in a video. E.g. recurrent neural network based methods compute a hidden state for every frame of the video using a recurrent weight matrix. Similarly, cluster-and-aggregate based methods such as NetVLAD, have a learnable clustering matrix which is used to assign soft-clusters to every frame in the video. Since these models look at every frame in the video, the number of floating point operations (FLOPs) is still large even though the memory footprint is small. We focus on building compute-efficient video classification models which process fewer frames and hence have less number of FLOPs. Similar to memory efficient models, we use the idea of distillation albeit in a different setting. Specifically, in our case, a compute-heavy teacher which looks at all the frames in the video is used to train a compute-efficient student which looks at only a small fraction of frames in the video. This is in contrast to a typical memory efficient Teacher-Student setting, wherein both the teacher and the student look at all the frames in the video but the student has fewer parameters. Our work thus complements the research on memory efficient video classification. We do an extensive evaluation with three types of models for video classification,viz.(i) recurrent models (ii) cluster-and-aggregate models and (iii) memory-efficient cluster-and-aggregate models and show that in each of these cases, a see-it-all teacher can be used to train a compute efficient see-very-little student. We show that the proposed student network can reduce the inference time by 30% and the number of FLOPs by approximately 90% with a negligible drop in the performance.