# Self-Supervised Representation Learning from Flow Equivariance

arXiv: https://arxiv.org/abs/2101.06553 · HTML: https://mlsanity.com/paper/2101.06553/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 55 references that are in the map.
- First posted on arXiv 16 Jan 2021.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [R-FCN: Object Detection via Region-based Fully Convolutional Networks](https://mlsanity.com/paper/1605.06409/) (2016)

## Built on by (most cited first, 28 of 28)

- [CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion](https://mlsanity.com/paper/2210.10716/) (2022)
- [Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data](https://mlsanity.com/paper/2203.16258/) (2022)
- [MST: Masked Self-Supervised Transformer for Visual Representation](https://mlsanity.com/paper/2106.05656/) (2021)
- [CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow](https://mlsanity.com/paper/2203.16896/) (2022)
- [Siamese Image Modeling for Self-Supervised Vision Representation Learning](https://mlsanity.com/paper/2206.01204/) (2022)
- [Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video](https://mlsanity.com/paper/2310.08584/) (2023)
- [Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels](https://mlsanity.com/paper/2107.13741/) (2021)
- [Dense Unsupervised Learning for Video Segmentation](https://mlsanity.com/paper/2111.06265/) (2021)
- [Self-supervised video pretraining yields robust and more human-aligned visual representations](https://mlsanity.com/paper/2210.06433/) (2022)
- [A Study of the Generalizability of Self-Supervised Representations](https://mlsanity.com/paper/2109.09150/) (2021)
- [Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?](https://mlsanity.com/paper/2409.13456/) (2024)
- [OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networks](https://mlsanity.com/paper/2404.14027/) (2024)
- [Comparing Correspondences: Video Prediction with Correspondence-wise Losses](https://mlsanity.com/paper/2104.09498/) (2021)
- [PooDLe: Pooled and dense self-supervised learning from naturalistic videos](https://mlsanity.com/paper/2408.11208/) (2024)
- [Doduo: Learning Dense Visual Correspondence from Unsupervised Semantic-Aware Flow](https://mlsanity.com/paper/2309.15110/) (2023)
- [MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learning](https://mlsanity.com/paper/2506.08694/) (2025)
- [Online Unsupervised Learning of Visual Representations and Categories](https://mlsanity.com/paper/2109.05675/) (2021)
- [Embedding Space Interpolation Beyond Mini-Batch, Beyond Pairs and Beyond Examples](https://mlsanity.com/paper/2311.05538/) (2023)
- [Pixel-level Correspondence for Self-Supervised Learning from Video](https://mlsanity.com/paper/2207.03866/) (2022)
- [MOFO: MOtion FOcused Self-Supervision for Video Understanding](https://mlsanity.com/paper/2308.12447/) (2023)
- [Improving Equivariance in State-of-the-Art Supervised Depth and Normal Predictors](https://mlsanity.com/paper/2309.16646/) (2023)
- [Equivariant Spatio-Temporal Self-Supervision for LiDAR Object Detection](https://mlsanity.com/paper/2404.11737/) (2024)
- [Teach me how to Interpolate a Myriad of Embeddings](https://mlsanity.com/paper/2206.14868/) (2022)
- [Midway Network: Learning Representations for Recognition and Motion from Latent Dynamics](https://mlsanity.com/paper/2510.05558/) (2025)
- [CYBORGS: Contrastively Bootstrapping Object Representations by Grounding in Segmentation](https://mlsanity.com/paper/2203.09343/) (2022)
- [Continual Learning of Conjugated Visual Representations through Higher-order Motion Flows](https://mlsanity.com/paper/2409.11441/) (2024)
- [Towards Stable Self-Supervised Object Representations in Unconstrained Egocentric Video](https://mlsanity.com/paper/2603.13912/) (2026)
- [Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners](https://mlsanity.com/paper/2604.26488/) (2026)

## Abstract (arXiv)

Self-supervised representation learning is able to learn semantically meaningful features; however, much of its recent success relies on multiple crops of an image with very few objects. Instead of learning view-invariant representation from simple images, humans learn representations in a complex world with changing scenes by observing object movement, deformation, pose variation, and ego motion. Motivated by this ability, we present a new self-supervised learning representation framework that can be directly deployed on a video stream of complex scenes with many moving objects. Our framework features a simple flow equivariance objective that encourages the network to predict the features of another frame by applying a flow transformation to the features of the current frame. Our representations, learned from high-resolution raw video, can be readily used for downstream tasks on static images. Readout experiments on challenging semantic segmentation, instance segmentation, and object detection benchmarks show that we are able to outperform representations obtained from previous state-of-the-art methods including SimCLR and BYOL.