# View while Moving: Efficient Video Recognition in Long-untrimmed Videos

arXiv: https://arxiv.org/abs/2308.04834 · HTML: https://mlsanity.com/paper/2308.04834/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 72 references that are in the map.
- First posted on arXiv 9 Aug 2023.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Long-Term Feature Banks for Detailed Video Understanding](https://mlsanity.com/paper/1812.05038/) (2018)
- [EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition](https://mlsanity.com/paper/1908.08498/) (2019)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [TDN: Temporal Difference Networks for Efficient Action Recognition](https://mlsanity.com/paper/2012.10071/) (2020)
- [Soft Actor-Critic for Discrete Action Settings](https://mlsanity.com/paper/1910.07207/) (2019)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [AdaFrame: Adaptive Frame Selection for Fast Video Recognition](https://mlsanity.com/paper/1811.12432/) (2018)
- [SCSampler: Sampling Salient Clips from Video for Efficient Action Recognition](https://mlsanity.com/paper/1904.04289/) (2019)
- [ECO: Efficient Convolutional Network for Online Video Understanding](https://mlsanity.com/paper/1804.09066/) (2018)
- [Cross-Modal and Hierarchical Modeling of Video and Text](https://mlsanity.com/paper/1810.07212/) (2018)
- [End-to-end Learning of Action Detection from Frame Glimpses in Videos](https://mlsanity.com/paper/1511.06984/) (2015)
- [TEINet: Towards an Efficient Architecture for Video Recognition](https://mlsanity.com/paper/1911.09435/) (2019)
- [AR-Net: Adaptive Frame Resolution for Efficient Action Recognition](https://mlsanity.com/paper/2007.15796/) (2020)
- [Long-term Temporal Convolutions for Action Recognition](https://mlsanity.com/paper/1604.04494/) (2016)
- [Adaptive Focus for Efficient Video Recognition](https://mlsanity.com/paper/2105.03245/) (2021)

## Built on by (most cited first, 4 of 4)

- [LongVLM: Efficient Long Video Understanding via Large Language Models](https://mlsanity.com/paper/2404.03384/) (2024)
- [Part-Attention Based Model Make Occluded Person Re-Identification Stronger](https://mlsanity.com/paper/2404.03443/) (2024)
- [EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception](https://mlsanity.com/paper/2506.21080/) (2025)
- [AdaViPro: Region-based Adaptive Visual Prompt for Large-Scale Models Adapting](https://mlsanity.com/paper/2403.13282/) (2024)

## Abstract (arXiv)

Recent adaptive methods for efficient video recognition mostly follow the two-stage paradigm of "preview-then-recognition" and have achieved great success on multiple video benchmarks. However, this two-stage paradigm involves two visits of raw frames from coarse-grained to fine-grained during inference (cannot be parallelized), and the captured spatiotemporal features cannot be reused in the second stage (due to varying granularity), being not friendly to efficiency and computation optimization. To this end, inspired by human cognition, we propose a novel recognition paradigm of "View while Moving" for efficient long-untrimmed video recognition. In contrast to the two-stage paradigm, our paradigm only needs to access the raw frame once. The two phases of coarse-grained sampling and fine-grained recognition are combined into unified spatiotemporal modeling, showing great performance. Moreover, we investigate the properties of semantic units in video and propose a hierarchical mechanism to efficiently capture and reason about the unit-level and video-level temporal semantics in long-untrimmed videos respectively. Extensive experiments on both long-untrimmed and short-trimmed videos demonstrate that our approach outperforms state-of-the-art methods in terms of accuracy as well as efficiency, yielding new efficiency and accuracy trade-offs for video spatiotemporal modeling.