# Dynamic Temporal Pyramid Network: A Closer Look at Multi-Scale Modeling for Activity Detection

arXiv: https://arxiv.org/abs/1808.02536 · HTML: https://mlsanity.com/paper/1808.02536/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2018.

## Built on by (most cited first, 8 of 8)

- [TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models](https://mlsanity.com/paper/2410.10818/) (2024)
- [MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsing](https://mlsanity.com/paper/2111.12374/) (2021)
- [Motion-driven Visual Tempo Learning for Video-based Action Recognition](https://mlsanity.com/paper/2202.12116/) (2022)
- [GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition](https://mlsanity.com/paper/2401.10039/) (2024)
- [Deep video representation learning: a survey](https://mlsanity.com/paper/2405.06574/) (2024)
- [TPDiff: Temporal Pyramid Video Diffusion Model](https://mlsanity.com/paper/2503.09566/) (2025)
- [Precondition and Effect Reasoning for Action Recognition](https://mlsanity.com/paper/2112.10057/) (2021)
- [Insights from Visual Cognition: Understanding Human Action Dynamics with Overall Glance and Refined Gaze Transformer](https://mlsanity.com/paper/2604.06783/) (2026)

## Abstract (arXiv)

Recognizing instances at different scales simultaneously is a fundamental challenge in visual detection problems. While spatial multi-scale modeling has been well studied in object detection, how to effectively apply a multi-scale architecture to temporal models for activity detection is still under-explored. In this paper, we identify three unique challenges that need to be specifically handled for temporal activity detection compared to its spatial counterpart. To address all these issues, we propose Dynamic Temporal Pyramid Network (DTPN), a new activity detection framework with a multi-scale pyramidal architecture featuring three novel designs: (1) We sample input video frames dynamically with varying frame per seconds (FPS) to construct a natural pyramidal input for video of an arbitrary length. (2) We design a two-branch multi-scale temporal feature hierarchy to deal with the inherent temporal scale variation of activity instances. (3) We further exploit the temporal context of activities by appropriately fusing multi-scale feature maps, and demonstrate that both local and global temporal contexts are important. By combining all these components into a uniform network, we end up with a single-shot activity detector involving single-pass inferencing and end-to-end training. Extensive experiments show that the proposed DTPN achieves state-of-the-art performance on the challenging ActvityNet dataset.