# Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction

arXiv: https://arxiv.org/abs/1811.11387 · HTML: https://mlsanity.com/paper/1811.11387/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2018.

## Built on by (most cited first, 30 of 71)

- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples](https://mlsanity.com/paper/2103.05905/) (2021)
- [TCLR: Temporal Contrastive Learning for Video Representation](https://mlsanity.com/paper/2101.07974/) (2021)
- [Self-Supervised Learning for Videos: A Survey](https://mlsanity.com/paper/2207.00419/) (2022)
- [Robust Audio-Visual Instance Discrimination](https://mlsanity.com/paper/2103.15916/) (2021)
- [Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures](https://mlsanity.com/paper/2307.15220/) (2023)
- [Self-Supervised Material and Texture Representation Learning for Remote Sensing Tasks](https://mlsanity.com/paper/2112.01715/) (2021)
- [Motion-aware Contrastive Video Representation Learning via Foreground-background Merging](https://mlsanity.com/paper/2109.15130/) (2021)
- [Self-Supervised Pillar Motion Learning for Autonomous Driving](https://mlsanity.com/paper/2104.08683/) (2021)
- [Time-Equivariant Contrastive Video Representation Learning](https://mlsanity.com/paper/2112.03624/) (2021)
- [Video Contrastive Learning with Global Context](https://mlsanity.com/paper/2108.02722/) (2021)
- [Masked Motion Encoding for Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2210.06096/) (2022)
- [ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency](https://mlsanity.com/paper/2106.02342/) (2021)
- [Self-supervised Video Representation Learning by Context and Motion Decoupling](https://mlsanity.com/paper/2104.00862/) (2021)
- [XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning](https://mlsanity.com/paper/2211.13929/) (2022)
- [Seeing the Arrow of Time in Large Multimodal Models](https://mlsanity.com/paper/2506.03340/) (2025)
- [Self-Supervised Video Representation Learning with Meta-Contrastive Network](https://mlsanity.com/paper/2108.08426/) (2021)
- [Domain Knowledge-Informed Self-Supervised Representations for Workout Form Assessment](https://mlsanity.com/paper/2202.14019/) (2022)
- [Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?](https://mlsanity.com/paper/2206.05266/) (2022)
- [Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works](https://mlsanity.com/paper/2211.08129/) (2022)
- [Motion-Focused Contrastive Learning of Video Representations](https://mlsanity.com/paper/2201.04029/) (2022)
- [How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?](https://mlsanity.com/paper/2203.14221/) (2022)
- [Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity](https://mlsanity.com/paper/2112.05883/) (2021)
- [Self-supervised Motion Learning from Static Images](https://mlsanity.com/paper/2104.00240/) (2021)
- [Long Short View Feature Decomposition via Contrastive Video Representation Learning](https://mlsanity.com/paper/2109.11593/) (2021)
- [Unsupervised Visual Representation Learning by Tracking Patches in Video](https://mlsanity.com/paper/2105.02545/) (2021)
- [Self-supervised Video Representation Learning with Motion-Aware Masked Autoencoders](https://mlsanity.com/paper/2210.04154/) (2022)
- [Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos](https://mlsanity.com/paper/2308.09951/) (2023)
- [Motion-Augmented Self-Training for Video Recognition at Smaller Scale](https://mlsanity.com/paper/2105.01646/) (2021)
- [TimeBalance: Temporally-Invariant and Temporally-Distinctive Video Representations for Semi-Supervised Action Recognition](https://mlsanity.com/paper/2303.16268/) (2023)

## Abstract (arXiv)

The success of deep neural networks generally requires a vast amount of training data to be labeled, which is expensive and unfeasible in scale, especially for video collections. To alleviate this problem, in this paper, we propose 3DRotNet: a fully self-supervised approach to learn spatiotemporal features from unlabeled videos. A set of rotations are applied to all videos, and a pretext task is defined as prediction of these rotations. When accomplishing this task, 3DRotNet is actually trained to understand the semantic concepts and motions in videos. In other words, it learns a spatiotemporal video representation, which can be transferred to improve video understanding tasks in small datasets. Our extensive experiments successfully demonstrate the effectiveness of the proposed framework on action recognition, leading to significant improvements over the state-of-the-art self-supervised methods. With the self-supervised pre-trained 3DRotNet from large datasets, the recognition accuracy is boosted up by 20.4% on UCF101 and 16.7% on HMDB51 respectively, compared to the models trained from scratch.