# Watching the World Go By: Representation Learning from Unlabeled Videos

arXiv: https://arxiv.org/abs/2003.07990 · HTML: https://mlsanity.com/paper/2003.07990/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Mar 2020.

## Built on by (most cited first, 30 of 59)

- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data](https://mlsanity.com/paper/2103.16607/) (2021)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound](https://mlsanity.com/paper/2201.02639/) (2022)
- [Divide and Contrast: Self-supervised Learning from Uncurated Data](https://mlsanity.com/paper/2105.08054/) (2021)
- [Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective](https://mlsanity.com/paper/2103.17263/) (2021)
- [Video Transformers: A Survey](https://mlsanity.com/paper/2201.05991/) (2022)
- [Semi-Supervised Action Recognition with Temporal Contrastive Learning](https://mlsanity.com/paper/2102.02751/) (2021)
- [Representation Learning via Global Temporal Alignment and Cycle-Consistency](https://mlsanity.com/paper/2105.05217/) (2021)
- [Time-Equivariant Contrastive Video Representation Learning](https://mlsanity.com/paper/2112.03624/) (2021)
- [SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving](https://mlsanity.com/paper/2106.11118/) (2021)
- [Video Contrastive Learning with Global Context](https://mlsanity.com/paper/2108.02722/) (2021)
- [Learning Pixel Trajectories with Multiscale Contrastive Random Walks](https://mlsanity.com/paper/2201.08379/) (2022)
- [Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations](https://mlsanity.com/paper/2308.11796/) (2023)
- [Joint Embedding Predictive Architectures Focus on Slow Features](https://mlsanity.com/paper/2211.10831/) (2022)
- [Self-Supervised Representation Learning from Flow Equivariance](https://mlsanity.com/paper/2101.06553/) (2021)
- [Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimization](https://mlsanity.com/paper/2108.02183/) (2021)
- [Motion-Focused Contrastive Learning of Video Representations](https://mlsanity.com/paper/2201.04029/) (2022)
- [DropMAE: Learning Representations via Masked Autoencoders with Spatial-Attention Dropout for Temporal Matching Tasks](https://mlsanity.com/paper/2304.00571/) (2023)
- [Instance Level Affinity-Based Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2104.01286/) (2021)
- [Self-supervised video pretraining yields robust and more human-aligned visual representations](https://mlsanity.com/paper/2210.06433/) (2022)
- [Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency](https://mlsanity.com/paper/2105.06463/) (2021)
- [Learning to Cut by Watching Movies](https://mlsanity.com/paper/2108.04294/) (2021)
- [Sound Localization by Self-Supervised Time Delay Estimation](https://mlsanity.com/paper/2204.12489/) (2022)
- [Modelling Neighbor Relation in Joint Space-Time Graph for Video Correspondence Learning](https://mlsanity.com/paper/2109.13499/) (2021)
- [Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision](https://mlsanity.com/paper/2112.05181/) (2021)
- [Learning from Untrimmed Videos: Self-Supervised Video Representation Learning with Hierarchical Consistency](https://mlsanity.com/paper/2204.03017/) (2022)
- [MemSAC: Memory Augmented Sample Consistency for Large Scale Unsupervised Domain Adaptation](https://mlsanity.com/paper/2207.12389/) (2022)
- [Time to augment self-supervised visual representation learning](https://mlsanity.com/paper/2207.13492/) (2022)
- [ViC-MAE: Self-Supervised Representation Learning from Images and Video with Contrastive Masked Autoencoders](https://mlsanity.com/paper/2303.12001/) (2023)

## Abstract (arXiv)

Recent single image unsupervised representation learning techniques show remarkable success on a variety of tasks. The basic principle in these works is instance discrimination: learning to differentiate between two augmented versions of the same image and a large batch of unrelated images. Networks learn to ignore the augmentation noise and extract semantically meaningful representations. Prior work uses artificial data augmentation techniques such as cropping, and color jitter which can only affect the image in superficial ways and are not aligned with how objects actually change e.g. occlusion, deformation, viewpoint change. In this paper, we argue that videos offer this natural augmentation for free. Videos can provide entirely new views of objects, show deformation, and even connect semantically similar but visually distinct concepts. We propose Video Noise Contrastive Estimation, a method for using unlabeled video to learn strong, transferable single image representations. We demonstrate improvements over recent unsupervised single image techniques, as well as over fully supervised ImageNet pretraining, across a variety of temporal and non-temporal tasks. Code and the Random Related Video Views dataset are available at https://www.github.com/danielgordon10/vince