# On the Integration of Optical Flow and Action Recognition

arXiv: https://arxiv.org/abs/1712.08416 · HTML: https://mlsanity.com/paper/1712.08416/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2017.

## Built on by (most cited first, 30 of 42)

- [CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow](https://mlsanity.com/paper/2203.16896/) (2022)
- [Full-Duplex Strategy for Video Object Segmentation](https://mlsanity.com/paper/2108.03151/) (2021)
- [DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videos](https://mlsanity.com/paper/2304.06544/) (2023)
- [Motion-Guided Masking for Spatiotemporal Representation Learning](https://mlsanity.com/paper/2308.12962/) (2023)
- [DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework](https://mlsanity.com/paper/2503.14880/) (2025)
- [A Survey on Video Action Recognition in Sports: Datasets, Methods and Applications](https://mlsanity.com/paper/2206.01038/) (2022)
- [DG-STGCN: Dynamic Spatial-Temporal Modeling for Skeleton-based Action Recognition](https://mlsanity.com/paper/2210.05895/) (2022)
- [LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation](https://mlsanity.com/paper/2306.08736/) (2023)
- [Biased Mixtures Of Experts: Enabling Computer Vision Inference Under Data Transfer Limitations](https://mlsanity.com/paper/2008.09662/) (2020)
- [Is Appearance Free Action Recognition Possible?](https://mlsanity.com/paper/2207.06261/) (2022)
- [TrouSPI-Net: Spatio-temporal attention on parallel atrous convolutions and U-GRUs for skeletal pedestrian crossing prediction](https://mlsanity.com/paper/2109.00953/) (2021)
- [Motion Sensitive Contrastive Learning for Self-supervised Video Representation](https://mlsanity.com/paper/2208.06105/) (2022)
- [Memory-Efficient Optical Flow via Radius-Distribution Orthogonal Cost Volume](https://mlsanity.com/paper/2312.03790/) (2023)
- [Review on Action Recognition for Accident Detection in Smart City Transportation Systems](https://mlsanity.com/paper/2208.09588/) (2022)
- [Scale-Adaptive Feature Aggregation for Efficient Space-Time Video Super-Resolution](https://mlsanity.com/paper/2310.17294/) (2023)
- [SplatFlow: Learning Multi-frame Optical Flow via Splatting](https://mlsanity.com/paper/2306.08887/) (2023)
- [Video Violence Recognition and Localization Using a Semi-Supervised Hard Attention Model](https://mlsanity.com/paper/2202.02212/) (2022)
- [Unsupervised Joint Learning of Depth, Optical Flow, Ego-motion from Video](https://mlsanity.com/paper/2105.14520/) (2021)
- [Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception](https://mlsanity.com/paper/2405.16493/) (2024)
- [ALBAR: Adversarial Learning approach to mitigate Biases in Action Recognition](https://mlsanity.com/paper/2502.00156/) (2025)
- [CLIP-FLow: Contrastive Learning by semi-supervised Iterative Pseudo labeling for Optical Flow Estimation](https://mlsanity.com/paper/2210.14383/) (2022)
- [Weakly-supervised Micro- and Macro-expression Spotting Based on Multi-level Consistency](https://mlsanity.com/paper/2305.02734/) (2023)
- [Adaptive Intermediate Representations for Video Understanding](https://mlsanity.com/paper/2104.07135/) (2021)
- [Unsupervised Flow Refinement near Motion Boundaries](https://mlsanity.com/paper/2208.02305/) (2022)
- [Attentive Multimodal Fusion for Optical and Scene Flow](https://mlsanity.com/paper/2307.15301/) (2023)
- [Adversarially Robust Video Perception by Seeing Motion](https://mlsanity.com/paper/2212.07815/) (2022)
- [A Multi-Modal Transformer Network for Action Detection](https://mlsanity.com/paper/2305.19624/) (2023)
- [Espresso: High Compression For Rich Extraction From Videos for Your Vision-Language Model](https://mlsanity.com/paper/2412.04729/) (2024)
- [MOOSE: Pay Attention to Temporal Dynamics for Video Understanding via Optical Flows](https://mlsanity.com/paper/2506.01119/) (2025)
- [It's a Matter of Time: Three Lessons on Long-Term Motion for Perception](https://mlsanity.com/paper/2602.14705/) (2026)

## Abstract (arXiv)

Most of the top performing action recognition methods use optical flow as a "black box" input. Here we take a deeper look at the combination of flow and action recognition, and investigate why optical flow is helpful, what makes a flow method good for action recognition, and how we can make it better. In particular, we investigate the impact of different flow algorithms and input transformations to better understand how these affect a state-of-the-art action recognition method. Furthermore, we fine tune two neural-network flow methods end-to-end on the most widely used action recognition dataset (UCF101). Based on these experiments, we make the following five observations: 1) optical flow is useful for action recognition because it is invariant to appearance, 2) optical flow methods are optimized to minimize end-point-error (EPE), but the EPE of current methods is not well correlated with action recognition performance, 3) for the flow methods tested, accuracy at boundaries and at small displacements is most correlated with action recognition performance, 4) training optical flow to minimize classification error instead of minimizing EPE improves recognition performance, and 5) optical flow learned for the task of action recognition differs from traditional optical flow especially inside the human body and at the boundary of the body. These observations may encourage optical flow researchers to look beyond EPE as a goal and guide action recognition researchers to seek better motion cues, leading to a tighter integration of the optical flow and action recognition communities.