# Self-Supervised Video Representation Learning with Meta-Contrastive Network

arXiv: https://arxiv.org/abs/2108.08426 · HTML: https://mlsanity.com/paper/2108.08426/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 59 references that are in the map.
- First posted on arXiv 19 Aug 2021.

## Builds on (most cited first, 30 of 46)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [Self-supervised Co-training for Video Representation Learning](https://mlsanity.com/paper/2010.09709/) (2020)
- [Unsupervised Representation Learning by Sorting Sequences](https://mlsanity.com/paper/1708.01246/) (2017)

## Built on by (most cited first, 28 of 28)

- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Self-supervised Video Transformer](https://mlsanity.com/paper/2112.01514/) (2021)
- [Robust Cross-Modal Representation Learning with Progressive Self-Distillation](https://mlsanity.com/paper/2204.04588/) (2022)
- [Probabilistic Representations for Video Contrastive Learning](https://mlsanity.com/paper/2204.03946/) (2022)
- [Text-Driven Image Editing via Learnable Regions](https://mlsanity.com/paper/2311.16432/) (2023)
- [How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?](https://mlsanity.com/paper/2203.14221/) (2022)
- [Learning to Align Sequential Actions in the Wild](https://mlsanity.com/paper/2111.09301/) (2021)
- [SMAUG: Sparse Masked Autoencoder for Efficient Video-Language Pre-training](https://mlsanity.com/paper/2211.11446/) (2022)
- [Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization](https://mlsanity.com/paper/2303.11003/) (2023)
- [Motion Sensitive Contrastive Learning for Self-supervised Video Representation](https://mlsanity.com/paper/2208.06105/) (2022)
- [It Takes Two: Masked Appearance-Motion Modeling for Self-supervised Video Transformer Pre-training](https://mlsanity.com/paper/2210.05234/) (2022)
- [Language-based Action Concept Spaces Improve Video Self-Supervised Learning](https://mlsanity.com/paper/2307.10922/) (2023)
- [Online Continual Learning via the Meta-learning Update with Multi-scale Knowledge Distillation and Data Augmentation](https://mlsanity.com/paper/2209.06107/) (2022)
- [Learning Transferable Spatiotemporal Representations from Natural Script Knowledge](https://mlsanity.com/paper/2209.15280/) (2022)
- [Harnessing Meta-Learning for Improving Full-Frame Video Stabilization](https://mlsanity.com/paper/2403.03662/) (2024)
- [Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception](https://mlsanity.com/paper/2405.16493/) (2024)
- [Bi-Calibration Networks for Weakly-Supervised Video Representation Learning](https://mlsanity.com/paper/2206.10491/) (2022)
- [SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learning](https://mlsanity.com/paper/2504.05706/) (2025)
- [Auxiliary Learning for Self-Supervised Video Representation via Similarity-based Knowledge Distillation](https://mlsanity.com/paper/2112.04011/) (2021)
- [Self-supervised Video Representation Learning with Cascade Positive Retrieval](https://mlsanity.com/paper/2201.07989/) (2022)
- [Harnessing Meta-Learning for Controllable Full-Frame Video Stabilization](https://mlsanity.com/paper/2508.18859/) (2025)
- [MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models](https://mlsanity.com/paper/2509.08538/) (2025)
- [GOCA: Guided Online Cluster Assignment for Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2207.10158/) (2022)
- [Online Continual Learning via the Knowledge Invariant and Spread-out Properties](https://mlsanity.com/paper/2302.00858/) (2023)
- [Reforming the Mechanism: Editing Reasoning Patterns in LLMs with Circuit Reshaping](https://mlsanity.com/paper/2603.06923/) (2026)
- [iBoot: Image-bootstrapped Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2206.08339/) (2022)
- [Collaboratively Self-supervised Video Representation Learning for Action Recognition](https://mlsanity.com/paper/2401.07584/) (2024)
- [Emulating Brain-like Rapid Learning in Neuromorphic Edge Computing](https://mlsanity.com/paper/2408.15800/) (2024)

## Abstract (arXiv)

Self-supervised learning has been successfully applied to pre-train video representations, which aims at efficient adaptation from pre-training domain to downstream tasks. Existing approaches merely leverage contrastive loss to learn instance-level discrimination. However, lack of category information will lead to hard-positive problem that constrains the generalization ability of this kind of methods. We find that the multi-task process of meta learning can provide a solution to this problem. In this paper, we propose a Meta-Contrastive Network (MCN), which combines the contrastive learning and meta learning, to enhance the learning ability of existing self-supervised approaches. Our method contains two training stages based on model-agnostic meta learning (MAML), each of which consists of a contrastive branch and a meta branch. Extensive evaluations demonstrate the effectiveness of our method. For two downstream tasks, i.e., video action recognition and video retrieval, MCN outperforms state-of-the-art approaches on UCF101 and HMDB51 datasets. To be more specific, with R(2+1)D backbone, MCN achieves Top-1 accuracies of 84.8% and 54.5% for video action recognition, as well as 52.5% and 23.7% for video retrieval.