# End-to-end Video-level Representation Learning for Action Recognition

arXiv: https://arxiv.org/abs/1711.04161 · HTML: https://mlsanity.com/paper/1711.04161/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2017.

## Built on by (most cited first, 4 of 4)

- [Deep Neural Networks in Video Human Action Recognition: A Review](https://mlsanity.com/paper/2305.15692/) (2023)
- [Cross-Block Fine-Grained Semantic Cascade for Skeleton-Based Sports Action Recognition](https://mlsanity.com/paper/2404.19383/) (2024)
- [VT-LVLM-AR: A Video-Temporal Large Vision-Language Model Adapter for Fine-Grained Action Recognition in Long-Term Videos](https://mlsanity.com/paper/2508.15903/) (2025)
- [Texture-Based Input Feature Selection for Action Recognition](https://mlsanity.com/paper/2303.00138/) (2023)

## Abstract (arXiv)

From the frame/clip-level feature learning to the video-level representation building, deep learning methods in action recognition have developed rapidly in recent years. However, current methods suffer from the confusion caused by partial observation training, or without end-to-end learning, or restricted to single temporal scale modeling and so on. In this paper, we build upon two-stream ConvNets and propose Deep networks with Temporal Pyramid Pooling (DTPP), an end-to-end video-level representation learning approach, to address these problems. Specifically, at first, RGB images and optical flow stacks are sparsely sampled across the whole video. Then a temporal pyramid pooling layer is used to aggregate the frame-level features which consist of spatial and temporal cues. Lastly, the trained model has compact video-level representation with multiple temporal scales, which is both global and sequence-aware. Experimental results show that DTPP achieves the state-of-the-art performance on two challenging video action datasets: UCF101 and HMDB51, either by ImageNet pre-training or Kinetics pre-training.