# Hidden Two-Stream Convolutional Networks for Action Recognition

arXiv: https://arxiv.org/abs/1704.00389 · HTML: https://mlsanity.com/paper/1704.00389/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2017.

## Built on by (most cited first, 24 of 24)

- [AIM: Adapting Image Models for Efficient Video Action Recognition](https://mlsanity.com/paper/2302.03024/) (2023)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [Video Contrastive Learning with Global Context](https://mlsanity.com/paper/2108.02722/) (2021)
- [Aligning Correlation Information for Domain Adaptation in Action Recognition](https://mlsanity.com/paper/2107.04932/) (2021)
- [A Large-scale Study of Spatiotemporal Representation Learning with a New Benchmark on Action Recognition](https://mlsanity.com/paper/2303.13505/) (2023)
- [A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data Modalities](https://mlsanity.com/paper/2408.05436/) (2024)
- [A Deep Learning-based Multimodal Depth-Aware Dynamic Hand Gesture Recognition System](https://mlsanity.com/paper/2107.02543/) (2021)
- [CNN-Based Action Recognition and Pose Estimation for Classifying Animal Behavior from Videos: A Survey](https://mlsanity.com/paper/2301.06187/) (2023)
- [Animal Behavior Analysis Methods Using Deep Learning: A Survey](https://mlsanity.com/paper/2405.14002/) (2024)
- [Unsupervised Motion Representation Enhanced Network for Action Recognition](https://mlsanity.com/paper/2103.03465/) (2021)
- [Video Action Recognition Using spatio-temporal optical flow video frames](https://mlsanity.com/paper/2103.05101/) (2021)
- [PLSM: A Parallelized Liquid State Machine for Unintentional Action Detection](https://mlsanity.com/paper/2105.09909/) (2021)
- [Can lies be faked? Comparing low-stakes and high-stakes deception video datasets from a Machine Learning perspective](https://mlsanity.com/paper/2211.13035/) (2022)
- [A Multi-Modal Transformer Network for Action Detection](https://mlsanity.com/paper/2305.19624/) (2023)
- [Temporal vs. Spatial: Comparing DINOv3 and V-JEPA2 Feature Representations for Video Action Analysis](https://mlsanity.com/paper/2509.21595/) (2025)
- [Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs](https://mlsanity.com/paper/2605.22823/) (2026)
- [DroneAttention: Sparse Weighted Temporal Attention for Drone-Camera Based Activity Recognition](https://mlsanity.com/paper/2212.03384/) (2022)
- [Beyond still images: Temporal features and input variance resilience](https://mlsanity.com/paper/2311.00800/) (2023)
- [Long-Short Temporal Modeling for Efficient Action Recognition](https://mlsanity.com/paper/2106.15787/) (2021)
- [Dynamic Appearance: A Video Representation for Action Recognition with Joint Training](https://mlsanity.com/paper/2211.12748/) (2022)
- [Enhancing Video Understanding: Deep Neural Networks for Spatiotemporal Analysis](https://mlsanity.com/paper/2502.07277/) (2025)
- [CA3D: Convolutional-Attentional 3D Nets for Efficient Video Activity Recognition on the Edge](https://mlsanity.com/paper/2505.19928/) (2025)
- [Improving action classification with brain-inspired deep networks](https://mlsanity.com/paper/2512.07729/) (2025)
- [WiFlow: Estimating Optical Flow using WiFi Channel State Information](https://mlsanity.com/paper/2609.02452/) (2026)

## Abstract (arXiv)

Analyzing videos of human actions involves understanding the temporal relationships among video frames. State-of-the-art action recognition approaches rely on traditional optical flow estimation methods to pre-compute motion information for CNNs. Such a two-stage approach is computationally expensive, storage demanding, and not end-to-end trainable. In this paper, we present a novel CNN architecture that implicitly captures motion information between adjacent frames. We name our approach hidden two-stream CNNs because it only takes raw video frames as input and directly predicts action classes without explicitly computing optical flow. Our end-to-end approach is 10x faster than its two-stage baseline. Experimental results on four challenging action recognition datasets: UCF101, HMDB51, THUMOS14 and ActivityNet v1.2 show that our approach significantly outperforms the previous best real-time approaches.