# Inter-intra Variant Dual Representations forSelf-supervised Video Recognition

arXiv: https://arxiv.org/abs/2107.01194 · HTML: https://mlsanity.com/paper/2107.01194/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 44 references that are in the map.
- First posted on arXiv 2 Jul 2021.

## Builds on (most cited first, 30 of 34)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Self-supervised Co-training for Video Representation Learning](https://mlsanity.com/paper/2010.09709/) (2020)
- [Unsupervised Representation Learning by Sorting Sequences](https://mlsanity.com/paper/1708.01246/) (2017)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [SpeedNet: Learning the Speediness in Videos](https://mlsanity.com/paper/2004.06130/) (2020)
- [VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples](https://mlsanity.com/paper/2103.05905/) (2021)
- [Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles](https://mlsanity.com/paper/1811.09795/) (2018)
- [Self-supervised Video Representation Learning by Pace Prediction](https://mlsanity.com/paper/2008.05861/) (2020)
- [Memory-augmented Dense Predictive Coding for Video Representation Learning](https://mlsanity.com/paper/2008.01065/) (2020)
- [Negative Margin Matters: Understanding Margin in Few-shot Classification](https://mlsanity.com/paper/2003.12060/) (2020)
- [Deep Representation Learning on Long-tailed Data: A Learnable Embedding Augmentation Perspective](https://mlsanity.com/paper/2002.10826/) (2020)
- [Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learning](https://mlsanity.com/paper/2006.11476/) (2020)
- [Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval](https://mlsanity.com/paper/2007.12163/) (2020)
- [Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics](https://mlsanity.com/paper/1904.03597/) (2019)
- [Video Representation Learning by Recognizing Temporal Transformations](https://mlsanity.com/paper/2007.10730/) (2020)
- [RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning](https://mlsanity.com/paper/2011.07949/) (2020)
- [Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning](https://mlsanity.com/paper/2001.00294/) (2020)
- [Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning](https://mlsanity.com/paper/2009.05769/) (2020)
- [ACTION-Net: Multipath Excitation for Action Recognition](https://mlsanity.com/paper/2103.07372/) (2021)
- [Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework](https://mlsanity.com/paper/2008.02531/) (2020)

## Built on by (most cited first, 5 of 5)

- [Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization](https://mlsanity.com/paper/2303.11003/) (2023)
- [C3: Cross-instance guided Contrastive Clustering](https://mlsanity.com/paper/2211.07136/) (2022)
- [LocoMotion: Learning Motion-Focused Video-Language Representations](https://mlsanity.com/paper/2410.12018/) (2024)
- [GOCA: Guided Online Cluster Assignment for Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2207.10158/) (2022)
- [Collaboratively Self-supervised Video Representation Learning for Action Recognition](https://mlsanity.com/paper/2401.07584/) (2024)

## Abstract (arXiv)

Contrastive learning applied to self-supervised representation learning has seen a resurgence in deep models. In this paper, we find that existing contrastive learning based solutions for self-supervised video recognition focus on inter-variance encoding but ignore the intra-variance existing in clips within the same video. We thus propose to learn dual representations for each clip which (romannumeral 1) encode intra-variance through a shuffle-rank pretext task; (romannumeral 2) encode inter-variance through a temporal coherent contrastive loss. Experiment results show that our method plays an essential role in balancing inter and intra variances and brings consistent performance gains on multiple backbones and contrastive learning frameworks. Integrated with SimCLR and pretrained on Kinetics-400, our method achieves 82.0% and 51.2% downstream classification accuracy on UCF101 and HMDB51 test sets respectively and 46.1% video retrieval accuracy on UCF101, outperforming both pretext-task based and contrastive learning based counterparts. Our code is available at hrefhttps://github.com/lzhangbj/DualVarhttps://github.com/lzhangbj/DualVar.