# Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition

arXiv: https://arxiv.org/abs/1711.11152 · HTML: https://mlsanity.com/paper/1711.11152/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2017.

## Built on by (most cited first, 30 of 82)

- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow](https://mlsanity.com/paper/2405.14793/) (2024)
- [FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation](https://mlsanity.com/paper/2303.01237/) (2023)
- [CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow](https://mlsanity.com/paper/2203.16896/) (2022)
- [VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation](https://mlsanity.com/paper/2303.08340/) (2023)
- [MemFlow: Optical Flow Estimation and Prediction with Memory](https://mlsanity.com/paper/2404.04808/) (2024)
- [E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition](https://mlsanity.com/paper/2112.03596/) (2021)
- [MGSampler: An Explainable Sampling Strategy for Video Action Recognition](https://mlsanity.com/paper/2104.09952/) (2021)
- [AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition](https://mlsanity.com/paper/2105.05165/) (2021)
- [Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognition](https://mlsanity.com/paper/2110.10101/) (2021)
- [WAFT: Warping-Alone Field Transforms for Optical Flow](https://mlsanity.com/paper/2506.21526/) (2025)
- [Rethinking Optical Flow from Geometric Matching Consistent Perspective](https://mlsanity.com/paper/2303.08384/) (2023)
- [Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection](https://mlsanity.com/paper/2108.03619/) (2021)
- [EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding](https://mlsanity.com/paper/2301.02217/) (2023)
- [Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition](https://mlsanity.com/paper/2102.07092/) (2021)
- [Mask Propagation for Efficient Video Semantic Segmentation](https://mlsanity.com/paper/2310.18954/) (2023)
- [No Pain, Big Gain: Classify Dynamic Point Cloud Sequences with Static Models by Fitting Feature-level Space-time Surfaces](https://mlsanity.com/paper/2203.11113/) (2022)
- [LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation](https://mlsanity.com/paper/2306.08736/) (2023)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [Hypergraph-based Multi-View Action Recognition using Event Cameras](https://mlsanity.com/paper/2403.19316/) (2024)
- [AnyFlow: Arbitrary Scale Optical Flow with Implicit Neural Representation](https://mlsanity.com/paper/2303.16493/) (2023)
- [RGB Stream Is Enough for Temporal Action Detection](https://mlsanity.com/paper/2107.04362/) (2021)
- [Relational Self-Attention: What's Missing in Attention for Video Understanding](https://mlsanity.com/paper/2111.01673/) (2021)
- [Gate-Shift-Fuse for Video Action Recognition](https://mlsanity.com/paper/2203.08897/) (2022)
- [Deformable Video Transformer](https://mlsanity.com/paper/2203.16795/) (2022)
- [SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model](https://mlsanity.com/paper/2307.16586/) (2023)
- [Deep Unsupervised Key Frame Extraction for Efficient Video Classification](https://mlsanity.com/paper/2211.06742/) (2022)
- [What Can Simple Arithmetic Operations Do for Temporal Modeling?](https://mlsanity.com/paper/2307.08908/) (2023)
- [Multi-Contextual Predictions with Vision Transformer for Video Anomaly Detection](https://mlsanity.com/paper/2206.08568/) (2022)
- [FlowSeek: Optical Flow Made Easier with Depth Foundation Models and Motion Bases](https://mlsanity.com/paper/2509.05297/) (2025)

## Abstract (arXiv)

Motion representation plays a vital role in human action recognition in videos. In this study, we introduce a novel compact motion representation for video action recognition, named Optical Flow guided Feature (OFF), which enables the network to distill temporal information through a fast and robust approach. The OFF is derived from the definition of optical flow and is orthogonal to the optical flow. The derivation also provides theoretical support for using the difference between two frames. By directly calculating pixel-wise spatiotemporal gradients of the deep feature maps, the OFF could be embedded in any existing CNN based video action recognition framework with only a slight additional cost. It enables the CNN to extract spatiotemporal information, especially the temporal information between frames simultaneously. This simple but powerful idea is validated by experimental results. The network with OFF fed only by RGB inputs achieves a competitive accuracy of 93.3% on UCF-101, which is comparable with the result obtained by two streams (RGB and optical flow), but is 15 times faster in speed. Experimental results also show that OFF is complementary to other motion modalities such as optical flow. When the proposed method is plugged into the state-of-the-art video action recognition framework, it has 96:0% and 74:2% accuracy on UCF-101 and HMDB-51 respectively. The code for this project is available at https://github.com/kevin-ssy/Optical-Flow-Guided-Feature.