# Hierarchical Contrastive Motion Learning for Video Action Recognition

arXiv: https://arxiv.org/abs/2007.10321 · HTML: https://mlsanity.com/paper/2007.10321/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2020.

## Built on by (most cited first, 8 of 8)

- [Self-Supervised Pillar Motion Learning for Autonomous Driving](https://mlsanity.com/paper/2104.08683/) (2021)
- [MGMAE: Motion Guided Masking for Video Masked Autoencoding](https://mlsanity.com/paper/2308.10794/) (2023)
- [ASM-Loc: Action-aware Segment Modeling for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2203.15187/) (2022)
- [Unsupervised Domain Adaptation for Video Transformers in Action Recognition](https://mlsanity.com/paper/2207.12842/) (2022)
- [Motion Aware Self-Supervision for Generic Event Boundary Detection](https://mlsanity.com/paper/2210.05574/) (2022)
- [MOFO: MOtion FOcused Self-Supervision for Video Understanding](https://mlsanity.com/paper/2308.12447/) (2023)
- [Ladder Siamese Network: a Method and Insights for Multi-level Self-Supervised Learning](https://mlsanity.com/paper/2211.13844/) (2022)
- [Domain Generalization for Improved Human Activity Recognition in Office Space Videos Using Adaptive Pre-processing](https://mlsanity.com/paper/2503.12678/) (2025)

## Abstract (arXiv)

One central question for video action recognition is how to model motion. In this paper, we present hierarchical contrastive motion learning, a new self-supervised learning framework to extract effective motion representations from raw video frames. Our approach progressively learns a hierarchy of motion features that correspond to different abstraction levels in a network. This hierarchical design bridges the semantic gap between low-level motion cues and high-level recognition tasks, and promotes the fusion of appearance and motion information at multiple levels. At each level, an explicit motion self-supervision is provided via contrastive learning to enforce the motion features at the current level to predict the future ones at the previous level. Thus, the motion features at higher levels are trained to gradually capture semantic dynamics and evolve more discriminative for action recognition. Our motion learning module is lightweight and flexible to be embedded into various backbone networks. Extensive experiments on four benchmarks show that the proposed approach consistently achieves superior results.