# Evolving Losses for Unsupervised Video Representation Learning

arXiv: https://arxiv.org/abs/2002.12177 · HTML: https://mlsanity.com/paper/2002.12177/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2020.

## Built on by (most cited first, 30 of 74)

- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?](https://mlsanity.com/paper/2106.11297/) (2021)
- [Broaden Your Views for Self-Supervised Video Learning](https://mlsanity.com/paper/2103.16559/) (2021)
- [Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers](https://mlsanity.com/paper/2103.16553/) (2021)
- [Robust Audio-Visual Instance Discrimination](https://mlsanity.com/paper/2103.15916/) (2021)
- [Bridging Video-text Retrieval with Multiple Choice Questions](https://mlsanity.com/paper/2201.04850/) (2022)
- [Self-supervised Video Transformer](https://mlsanity.com/paper/2112.01514/) (2021)
- [Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos](https://mlsanity.com/paper/2104.12671/) (2021)
- [Robust Cross-Modal Representation Learning with Progressive Self-Distillation](https://mlsanity.com/paper/2204.04588/) (2022)
- [Automated Self-Supervised Learning for Graphs](https://mlsanity.com/paper/2106.05470/) (2021)
- [Learning Cross-modal Contrastive Features for Video Domain Adaptation](https://mlsanity.com/paper/2108.11974/) (2021)
- [LiRA: Learning Visual Speech Representations from Audio through Self-supervision](https://mlsanity.com/paper/2106.09171/) (2021)
- [Video Contrastive Learning with Global Context](https://mlsanity.com/paper/2108.02722/) (2021)
- [VPN++: Rethinking Video-Pose embeddings for understanding Activities of Daily Living](https://mlsanity.com/paper/2105.08141/) (2021)
- [Self-supervised Video Representation Learning by Context and Motion Decoupling](https://mlsanity.com/paper/2104.00862/) (2021)
- [Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning](https://mlsanity.com/paper/2103.10211/) (2021)
- [TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2112.03587/) (2021)
- [XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning](https://mlsanity.com/paper/2211.13929/) (2022)
- [Self-supervised object detection from audio-visual correspondence](https://mlsanity.com/paper/2104.06401/) (2021)
- [MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval](https://mlsanity.com/paper/2204.12408/) (2022)
- [Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data](https://mlsanity.com/paper/2206.02353/) (2022)
- [How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?](https://mlsanity.com/paper/2203.14221/) (2022)
- [Static and Dynamic Concepts for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2207.12795/) (2022)
- [Learning State-Aware Visual Representations from Audible Interactions](https://mlsanity.com/paper/2209.13583/) (2022)
- [Long Short View Feature Decomposition via Contrastive Video Representation Learning](https://mlsanity.com/paper/2109.11593/) (2021)

## Abstract (arXiv)

We present a new method to learn video representations from large-scale unlabeled video data. Ideally, this representation will be generic and transferable, directly usable for new tasks such as action recognition and zero or few-shot learning. We formulate unsupervised representation learning as a multi-modal, multi-task learning problem, where the representations are shared across different modalities via distillation. Further, we introduce the concept of loss function evolution by using an evolutionary search algorithm to automatically find optimal combination of loss functions capturing many (self-supervised) tasks and modalities. Thirdly, we propose an unsupervised representation evaluation metric using distribution matching to a large unlabeled dataset as a prior constraint, based on Zipf's law. This unsupervised constraint, which is not guided by any labeling, produces similar results to weakly-supervised, task-specific ones. The proposed unsupervised representation learning results in a single RGB network and outperforms previous methods. Notably, it is also more effective than several label-based methods (e.g., ImageNet), with the exception of large, fully labeled video datasets.