# ActionFlowNet: Learning Motion Representation for Action Recognition

arXiv: https://arxiv.org/abs/1612.03052 · HTML: https://mlsanity.com/paper/1612.03052/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2016.

## Built on by (most cited first, 18 of 18)

- [MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition](https://mlsanity.com/paper/2108.09322/) (2021)
- [Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos](https://mlsanity.com/paper/2308.09951/) (2023)
- [End-to-End Compressed Video Representation Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2203.15336/) (2022)
- [FAR: Fourier Aerial Video Recognition](https://mlsanity.com/paper/2203.10694/) (2022)
- [Differentiable Frequency-based Disentanglement for Aerial Video Action Recognition](https://mlsanity.com/paper/2209.09194/) (2022)
- [CNN-Based Action Recognition and Pose Estimation for Classifying Animal Behavior from Videos: A Survey](https://mlsanity.com/paper/2301.06187/) (2023)
- [Flow-based Spatio-Temporal Structured Prediction of Motion Dynamics](https://mlsanity.com/paper/2104.04391/) (2021)
- [Adaptive Intermediate Representations for Video Understanding](https://mlsanity.com/paper/2104.07135/) (2021)
- [Local Compressed Video Stream Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2309.15431/) (2023)
- [Unsupervised Motion Representation Enhanced Network for Action Recognition](https://mlsanity.com/paper/2103.03465/) (2021)
- [PLSM: A Parallelized Liquid State Machine for Unintentional Action Detection](https://mlsanity.com/paper/2105.09909/) (2021)
- [Adversarially Robust Video Perception by Seeing Motion](https://mlsanity.com/paper/2212.07815/) (2022)
- [Spatio-Temporal Video Representation Learning for AI Based Video Playback Style Prediction](https://mlsanity.com/paper/2110.01015/) (2021)
- [DroneAttention: Sparse Weighted Temporal Attention for Drone-Camera Based Activity Recognition](https://mlsanity.com/paper/2212.03384/) (2022)
- [Exploring High-Order Self-Similarity for Video Understanding](https://mlsanity.com/paper/2604.20760/) (2026)
- [Dynamic Appearance: A Video Representation for Action Recognition with Joint Training](https://mlsanity.com/paper/2211.12748/) (2022)
- [Texture-Based Input Feature Selection for Action Recognition](https://mlsanity.com/paper/2303.00138/) (2023)
- [CA3D: Convolutional-Attentional 3D Nets for Efficient Video Activity Recognition on the Edge](https://mlsanity.com/paper/2505.19928/) (2025)

## Abstract (arXiv)

Even with the recent advances in convolutional neural networks (CNN) in various visual recognition tasks, the state-of-the-art action recognition system still relies on hand crafted motion feature such as optical flow to achieve the best performance. We propose a multitask learning model ActionFlowNet to train a single stream network directly from raw pixels to jointly estimate optical flow while recognizing actions with convolutional neural networks, capturing both appearance and motion in a single model. We additionally provide insights to how the quality of the learned optical flow affects the action recognition. Our model significantly improves action recognition accuracy by a large margin 31% compared to state-of-the-art CNN-based action recognition models trained without external large scale data and additional optical flow input. Without pretraining on large external labeled datasets, our model, by well exploiting the motion information, achieves competitive recognition accuracy to the models trained with large labeled datasets such as ImageNet and Sport-1M.