# Self-Supervised Learning of Video-Induced Visual Invariances

arXiv: https://arxiv.org/abs/1912.02783 · HTML: https://mlsanity.com/paper/1912.02783/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2019.

## Built on by (most cited first, 29 of 29)

- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [Towards Causal Representation Learning](https://mlsanity.com/paper/2102.11107/) (2021)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders](https://mlsanity.com/paper/2311.00566/) (2023)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation](https://mlsanity.com/paper/2112.09645/) (2021)
- [Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video](https://mlsanity.com/paper/2310.08584/) (2023)
- [Self-Supervised Learning for Semi-Supervised Temporal Action Proposal](https://mlsanity.com/paper/2104.03214/) (2021)
- [Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positives](https://mlsanity.com/paper/2201.11736/) (2022)
- [Shot Contrastive Self-Supervised Learning for Scene Boundary Detection](https://mlsanity.com/paper/2104.13537/) (2021)
- [Learning Representational Invariances for Data-Efficient Action Recognition](https://mlsanity.com/paper/2103.16565/) (2021)
- [Self-supervised video pretraining yields robust and more human-aligned visual representations](https://mlsanity.com/paper/2210.06433/) (2022)
- [3D Object Detection with a Self-supervised Lidar Scene Flow Backbone](https://mlsanity.com/paper/2205.00705/) (2022)
- [Learning to Cut by Watching Movies](https://mlsanity.com/paper/2108.04294/) (2021)
- [Object Priors for Classifying and Localizing Unseen Actions](https://mlsanity.com/paper/2104.04715/) (2021)
- [Composable Augmentation Encoding for Video Representation Learning](https://mlsanity.com/paper/2104.00616/) (2021)
- [Understanding Hyperbolic Metric Learning through Hard Negative Sampling](https://mlsanity.com/paper/2404.15523/) (2024)
- [Self-supervised visual learning from interactions with objects](https://mlsanity.com/paper/2407.06704/) (2024)
- [Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting](https://mlsanity.com/paper/2106.10137/) (2021)
- [A Systems Theory of Transfer Learning](https://mlsanity.com/paper/2107.01196/) (2021)
- [Self-Supervised Graph Transformer for Deepfake Detection](https://mlsanity.com/paper/2307.15019/) (2023)
- [Self-supervised Learning with Local Attention-Aware Feature](https://mlsanity.com/paper/2108.00475/) (2021)
- [MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learning](https://mlsanity.com/paper/2506.08694/) (2025)
- [A Computational Account Of Self-Supervised Visual Learning From Egocentric Object Play](https://mlsanity.com/paper/2305.19445/) (2023)
- [Pixel-level Correspondence for Self-Supervised Learning from Video](https://mlsanity.com/paper/2207.03866/) (2022)
- [Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification](https://mlsanity.com/paper/2408.14441/) (2024)
- [Learning to See Through a Baby's Eyes: Early Visual Diets Enable Robust Visual Intelligence in Humans and Machines](https://mlsanity.com/paper/2511.14440/) (2025)
- [Time-based Self-supervised Learning for Wireless Capsule Endoscopy](https://mlsanity.com/paper/2204.09773/) (2022)
- [Brain-Aligned Multi-Stream Video Transformers with Sparse Self-Selection](https://mlsanity.com/paper/2607.17625/) (2026)

## Abstract (arXiv)

We propose a general framework for self-supervised learning of transferable visual representations based on Video-Induced Visual Invariances (VIVI). We consider the implicit hierarchy present in the videos and make use of (i) frame-level invariances (e.g. stability to color and contrast perturbations), (ii) shot/clip-level invariances (e.g. robustness to changes in object orientation and lighting conditions), and (iii) video-level invariances (semantic relationships of scenes across shots/clips), to define a holistic self-supervised loss. Training models using different variants of the proposed framework on videos from the YouTube-8M (YT8M) data set, we obtain state-of-the-art self-supervised transfer learning results on the 19 diverse downstream tasks of the Visual Task Adaptation Benchmark (VTAB), using only 1000 labels per task. We then show how to co-train our models jointly with labeled images, outperforming an ImageNet-pretrained ResNet-50 by 0.8 points with 10x fewer labeled images, as well as the previous best supervised model by 3.7 points using the full ImageNet data set.