# Self-supervised Co-training for Video Representation Learning

arXiv: https://arxiv.org/abs/2010.09709 · HTML: https://mlsanity.com/paper/2010.09709/

## Key facts

- Cited by 232 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2020.

## Built on by (most cited first, 30 of 232)

- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [Balanced Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2207.09052/) (2022)
- [TCLR: Temporal Contrastive Learning for Video Representation](https://mlsanity.com/paper/2101.07974/) (2021)
- [V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning](https://mlsanity.com/paper/2603.14482/) (2026)
- [Broaden Your Views for Self-Supervised Video Learning](https://mlsanity.com/paper/2103.16559/) (2021)
- [3D Human Action Representation Learning via Cross-View Consistency Pursuit](https://mlsanity.com/paper/2104.14466/) (2021)
- [In Defense of Online Models for Video Instance Segmentation](https://mlsanity.com/paper/2207.10661/) (2022)
- [CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP](https://mlsanity.com/paper/2110.11316/) (2021)
- [Self-Supervised Learning for Videos: A Survey](https://mlsanity.com/paper/2207.00419/) (2022)
- [Temporal Alignment Networks for Long-term Video](https://mlsanity.com/paper/2204.02968/) (2022)
- [CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow](https://mlsanity.com/paper/2203.16896/) (2022)
- [Robust Audio-Visual Instance Discrimination](https://mlsanity.com/paper/2103.15916/) (2021)
- [Object discovery and representation networks](https://mlsanity.com/paper/2203.08777/) (2022)
- [Bridging Video-text Retrieval with Multiple Choice Questions](https://mlsanity.com/paper/2201.04850/) (2022)
- [CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations](https://mlsanity.com/paper/2109.14910/) (2021)
- [Dual Contrastive Learning for Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2104.07689/) (2021)
- [Dissecting Supervised Contrastive Learning](https://mlsanity.com/paper/2102.08817/) (2021)
- [Self-supervised Video Transformer](https://mlsanity.com/paper/2112.01514/) (2021)
- [Graph Contrastive Clustering](https://mlsanity.com/paper/2104.01429/) (2021)
- [MomentDiff: Generative Video Moment Retrieval from Random to Real](https://mlsanity.com/paper/2307.02869/) (2023)
- [Home Action Genome: Cooperative Compositional Action Understanding](https://mlsanity.com/paper/2105.05226/) (2021)
- [Equivariant Similarity for Vision-Language Foundation Models](https://mlsanity.com/paper/2303.14465/) (2023)

## Abstract (arXiv)

The objective of this paper is visual-only self-supervised video representation learning. We make the following contributions: (i) we investigate the benefit of adding semantic-class positives to instance-based Info Noise Contrastive Estimation (InfoNCE) training, showing that this form of supervised contrastive learning leads to a clear improvement in performance; (ii) we propose a novel self-supervised co-training scheme to improve the popular infoNCE loss, exploiting the complementary information from different views, RGB streams and optical flow, of the same data source by using one view to obtain positive class samples for the other; (iii) we thoroughly evaluate the quality of the learnt representation on two different downstream tasks: action recognition and video retrieval. In both cases, the proposed approach demonstrates state-of-the-art or comparable performance with other self-supervised approaches, whilst being significantly more efficient to train, i.e. requiring far less training data to achieve similar performance.