# ECO: Efficient Convolutional Network for Online Video Understanding

arXiv: https://arxiv.org/abs/1804.09066 · HTML: https://mlsanity.com/paper/1804.09066/

## Key facts

- Cited by 115 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2018.

## Built on by (most cited first, 30 of 115)

- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2206.13559/) (2022)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [Frozen CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2208.03550/) (2022)
- [BEVT: BERT Pretraining of Video Transformers](https://mlsanity.com/paper/2112.01529/) (2021)
- [Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition](https://mlsanity.com/paper/2501.03230/) (2024)
- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?](https://mlsanity.com/paper/2106.11297/) (2021)
- [Long Movie Clip Classification with State-Space Video Models](https://mlsanity.com/paper/2204.01692/) (2022)
- [An Image is Worth 16x16 Words, What is a Video Worth?](https://mlsanity.com/paper/2103.13915/) (2021)
- [VindLU: A Recipe for Effective Video-and-Language Pretraining](https://mlsanity.com/paper/2212.05051/) (2022)
- [Adaptive Focus for Efficient Video Recognition](https://mlsanity.com/paper/2105.03245/) (2021)
- [HierVL: Learning Hierarchical Video-Language Embeddings](https://mlsanity.com/paper/2301.02311/) (2023)
- [ACTION-Net: Multipath Excitation for Action Recognition](https://mlsanity.com/paper/2103.07372/) (2021)
- [FrameExit: Conditional Early Exiting for Efficient Video Recognition](https://mlsanity.com/paper/2104.13400/) (2021)
- [Efficient Video Transformers with Spatial-Temporal Token Selection](https://mlsanity.com/paper/2111.11591/) (2021)
- [MGSampler: An Explainable Sampling Strategy for Video Action Recognition](https://mlsanity.com/paper/2104.09952/) (2021)
- [AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition](https://mlsanity.com/paper/2102.05775/) (2021)
- [Don't Look Twice: Faster Video Transformers with Run-Length Tokenization](https://mlsanity.com/paper/2411.05222/) (2024)
- [Glance and Focus Networks for Dynamic Visual Recognition](https://mlsanity.com/paper/2201.03014/) (2022)
- [AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition](https://mlsanity.com/paper/2112.14238/) (2021)
- [Dynamic Perceiver for Efficient Visual Recognition](https://mlsanity.com/paper/2306.11248/) (2023)
- [Self-Regulated Learning for Egocentric Video Activity Anticipation](https://mlsanity.com/paper/2111.11631/) (2021)
- [EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding](https://mlsanity.com/paper/2301.02217/) (2023)
- [Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition](https://mlsanity.com/paper/2102.07092/) (2021)
- [Coarse-Fine Networks for Temporal Activity Detection in Videos](https://mlsanity.com/paper/2103.01302/) (2021)
- [CT-Net: Channel Tensorization Network for Video Classification](https://mlsanity.com/paper/2106.01603/) (2021)
- [No frame left behind: Full Video Action Recognition](https://mlsanity.com/paper/2103.15395/) (2021)
- [Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation](https://mlsanity.com/paper/2308.04549/) (2023)
- [OCSampler: Compressing Videos to One Clip with Single-step Sampling](https://mlsanity.com/paper/2201.04388/) (2022)

## Abstract (arXiv)

The state of the art in video understanding suffers from two problems: (1) The major part of reasoning is performed locally in the video, therefore, it misses important relationships within actions that span several seconds. (2) While there are local methods with fast per-frame processing, the processing of the whole video is not efficient and hampers fast video retrieval or online classification of long-term activities. In this paper, we introduce a network architecture that takes long-term content into account and enables fast per-video processing at the same time. The architecture is based on merging long-term content already in the network rather than in a post-hoc fusion. Together with a sampling strategy, which exploits that neighboring frames are largely redundant, this yields high-quality action classification and video captioning at up to 230 videos per second, where each video can consist of a few hundred frames. The approach achieves competitive performance across all datasets while being 10x to 80x faster than state-of-the-art methods.