# Condensing a Sequence to One Informative Frame for Video Recognition

arXiv: https://arxiv.org/abs/2201.04022 · HTML: https://mlsanity.com/paper/2201.04022/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 11 Jan 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Beyond Short Snippets: Deep Networks for Video Classification](https://mlsanity.com/paper/1503.08909/) (2015)
- [STM: SpatioTemporal and Motion Encoding for Action Recognition](https://mlsanity.com/paper/1908.02486/) (2019)
- [Compressed Video Action Recognition](https://mlsanity.com/paper/1712.00636/) (2017)
- [SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning](https://mlsanity.com/paper/2008.00975/) (2020)
- [Learning Spatio-Temporal Representation with Local and Global Diffusion](https://mlsanity.com/paper/1906.05571/) (2019)
- [A Multigrid Method for Efficiently Training Video Models](https://mlsanity.com/paper/1912.00998/) (2019)
- [Real-time Action Recognition with Enhanced Motion Vector CNNs](https://mlsanity.com/paper/1604.07669/) (2016)
- [Omni-sourced Webly-supervised Learning for Video Recognition](https://mlsanity.com/paper/2003.13042/) (2020)
- [Deep Temporal Linear Encoding Networks](https://mlsanity.com/paper/1611.06678/) (2016)
- [Action recognition with spatial-temporal discriminative filter banks](https://mlsanity.com/paper/1908.07625/) (2019)
- [Deep Quantization: Encoding Convolutional Activations with Deep Generative Model](https://mlsanity.com/paper/1611.09502/) (2016)
- [Video Representation Learning Using Discriminative Pooling](https://mlsanity.com/paper/1803.10628/) (2018)
- [AVD: Adversarial Video Distillation](https://mlsanity.com/paper/1907.05640/) (2019)

## Built on by (most cited first, 6 of 6)

- [Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning](https://mlsanity.com/paper/2211.00833/) (2022)
- [MLP-3D: A MLP-like 3D Architecture with Grouped Time Mixing](https://mlsanity.com/paper/2206.06292/) (2022)
- [Vision-Language Models Learn Super Images for Efficient Partially Relevant Video Retrieval](https://mlsanity.com/paper/2312.00414/) (2023)
- [Efficient Cross-Modal Video Retrieval with Meta-Optimized Frames](https://mlsanity.com/paper/2210.08452/) (2022)
- [Harnessing Object Grounding for Time-Sensitive Video Understanding](https://mlsanity.com/paper/2509.06335/) (2025)
- [Image Classifiers are Efficient Self-Supervised Video Representation Learners](https://mlsanity.com/paper/2609.40347/) (2026)

## Abstract (arXiv)

Video is complex due to large variations in motion and rich content in fine-grained visual details. Abstracting useful information from such information-intensive media requires exhaustive computing resources. This paper studies a two-step alternative that first condenses the video sequence to an informative "frame" and then exploits off-the-shelf image recognition system on the synthetic frame. A valid question is how to define "useful information" and then distill it from a video sequence down to one synthetic frame. This paper presents a novel Informative Frame Synthesis (IFS) architecture that incorporates three objective tasks, i.e., appearance reconstruction, video categorization, motion estimation, and two regularizers, i.e., adversarial learning, color consistency. Each task equips the synthetic frame with one ability, while each regularizer enhances its visual quality. With these, by jointly learning the frame synthesis in an end-to-end manner, the generated frame is expected to encapsulate the required spatio-temporal information useful for video analysis. Extensive experiments are conducted on the large-scale Kinetics dataset. When comparing to baseline methods that map video sequence to a single image, IFS shows superior performance. More remarkably, IFS consistently demonstrates evident improvements on image-based 2D networks and clip-based 3D networks, and achieves comparable performance with the state-of-the-art methods with less computational cost.