# End-to-End Learning of Motion Representation for Video Understanding

arXiv: https://arxiv.org/abs/1804.00413 · HTML: https://mlsanity.com/paper/1804.00413/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2018.

## Built on by (most cited first, 30 of 43)

- [Targeted Supervised Contrastive Learning for Long-Tailed Recognition](https://mlsanity.com/paper/2111.13998/) (2021)
- [MGSampler: An Explainable Sampling Strategy for Video Action Recognition](https://mlsanity.com/paper/2104.09952/) (2021)
- [MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition](https://mlsanity.com/paper/2108.09322/) (2021)
- [Disentangling Architecture and Training for Optical Flow](https://mlsanity.com/paper/2203.10712/) (2022)
- [Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition](https://mlsanity.com/paper/2102.07092/) (2021)
- [Graph Convolutional Module for Temporal Action Localization in Videos](https://mlsanity.com/paper/2112.00302/) (2021)
- [Action Sensitivity Learning for Temporal Action Localization](https://mlsanity.com/paper/2305.15701/) (2023)
- [Learning a Model-Driven Variational Network for Deformable Image Registration](https://mlsanity.com/paper/2105.12227/) (2021)
- [No Pain, Big Gain: Classify Dynamic Point Cloud Sequences with Static Models by Fitting Feature-level Space-time Surfaces](https://mlsanity.com/paper/2203.11113/) (2022)
- [Detector-Free Weakly Supervised Group Activity Recognition](https://mlsanity.com/paper/2204.02139/) (2022)
- [Learning to Refactor Action and Co-occurrence Features for Temporal Action Localization](https://mlsanity.com/paper/2206.11493/) (2022)
- [End-to-End Compressed Video Representation Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2203.15336/) (2022)
- [Relational Self-Attention: What's Missing in Attention for Video Understanding](https://mlsanity.com/paper/2111.01673/) (2021)
- [Gate-Shift-Fuse for Video Action Recognition](https://mlsanity.com/paper/2203.08897/) (2022)
- [Optical Flow Training under Limited Label Budget via Active Learning](https://mlsanity.com/paper/2203.05053/) (2022)
- [Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction](https://mlsanity.com/paper/2112.02252/) (2021)
- [DRO: Deep Recurrent Optimizer for Video to Depth](https://mlsanity.com/paper/2103.13201/) (2021)
- [Motion-driven Visual Tempo Learning for Video-based Action Recognition](https://mlsanity.com/paper/2202.12116/) (2022)
- [ADFactory: An Effective Framework for Generalizing Optical Flow with Nerf](https://mlsanity.com/paper/2311.04246/) (2023)
- [LAD: A Hybrid Deep Learning System for Benign Paroxysmal Positional Vertigo Disorders Diagnostic](https://mlsanity.com/paper/2210.08282/) (2022)
- [Adaptive Intermediate Representations for Video Understanding](https://mlsanity.com/paper/2104.07135/) (2021)
- [Motion Aware Self-Supervision for Generic Event Boundary Detection](https://mlsanity.com/paper/2210.05574/) (2022)
- [Coupling Global Context and Local Contents for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2304.09059/) (2023)
- [Local Compressed Video Stream Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2309.15431/) (2023)
- [Self-Assessed Generation: Trustworthy Label Generation for Optical Flow and Stereo Matching in Real-world](https://mlsanity.com/paper/2410.10453/) (2024)
- [Progressive Limb-Aware Virtual Try-On](https://mlsanity.com/paper/2503.12588/) (2025)
- [Unsupervised Motion Representation Enhanced Network for Action Recognition](https://mlsanity.com/paper/2103.03465/) (2021)
- [Busy-Quiet Video Disentangling for Video Classification](https://mlsanity.com/paper/2103.15584/) (2021)
- [BabyNet: A Lightweight Network for Infant Reaching Action Recognition in Unconstrained Environments to Support Future Pediatric Rehabilitation Applications](https://mlsanity.com/paper/2208.04950/) (2022)
- [ADU-Depth: Attention-based Distillation with Uncertainty Modeling for Depth Estimation](https://mlsanity.com/paper/2309.14744/) (2023)

## Abstract (arXiv)

Despite the recent success of end-to-end learned representations, hand-crafted optical flow features are still widely used in video analysis tasks. To fill this gap, we propose TVNet, a novel end-to-end trainable neural network, to learn optical-flow-like features from data. TVNet subsumes a specific optical flow solver, the TV-L1 method, and is initialized by unfolding its optimization iterations as neural layers. TVNet can therefore be used directly without any extra learning. Moreover, it can be naturally concatenated with other task-specific networks to formulate an end-to-end architecture, thus making our method more efficient than current multi-stage approaches by avoiding the need to pre-compute and store features on disk. Finally, the parameters of the TVNet can be further fine-tuned by end-to-end training. This enables TVNet to learn richer and task-specific patterns beyond exact optical flow. Extensive experiments on two action recognition benchmarks verify the effectiveness of the proposed approach. Our TVNet achieves better accuracies than all compared methods, while being competitive with the fastest counterpart in terms of features extraction time.