# On Compositions of Transformations in Contrastive Self-Supervised Learning

arXiv: https://arxiv.org/abs/2003.04298 · HTML: https://mlsanity.com/paper/2003.04298/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2020.

## Built on by (most cited first, 30 of 34)

- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [Vision Transformers are Parameter-Efficient Audio-Visual Learners](https://mlsanity.com/paper/2212.07983/) (2022)
- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [MAViL: Masked Audio-Video Learners](https://mlsanity.com/paper/2212.08071/) (2022)
- [Audiovisual Masked Autoencoders](https://mlsanity.com/paper/2212.05922/) (2022)
- [Robust Contrastive Learning against Noisy Views](https://mlsanity.com/paper/2201.04309/) (2022)
- [ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Genetics](https://mlsanity.com/paper/2111.13424/) (2021)
- [Versatile Multi-Modal Pre-Training for Human-Centric Perception](https://mlsanity.com/paper/2203.13815/) (2022)
- [TeD-SPAD: Temporal Distinctiveness for Self-supervised Privacy-preservation for video Anomaly Detection](https://mlsanity.com/paper/2308.11072/) (2023)
- [Learning Representational Invariances for Data-Efficient Action Recognition](https://mlsanity.com/paper/2103.16565/) (2021)
- [SelfD: Self-Learning Large-Scale Driving Policies From the Web](https://mlsanity.com/paper/2204.10320/) (2022)
- [CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment](https://mlsanity.com/paper/2505.01237/) (2025)
- [Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision](https://mlsanity.com/paper/2112.05181/) (2021)
- [STEPs: Self-Supervised Key Step Extraction and Localization from Unlabeled Procedural Videos](https://mlsanity.com/paper/2301.00794/) (2023)
- [SERE: Exploring Feature Self-relation for Self-supervised Transformer](https://mlsanity.com/paper/2206.05184/) (2022)
- [EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning](https://mlsanity.com/paper/2403.09502/) (2024)
- [Motion Sensitive Contrastive Learning for Self-supervised Video Representation](https://mlsanity.com/paper/2208.06105/) (2022)
- [TransRank: Self-supervised Video Representation Learning via Ranking-based Transformation Recognition](https://mlsanity.com/paper/2205.02028/) (2022)
- [Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicity](https://mlsanity.com/paper/2111.05329/) (2021)
- [No Free Lunch in Self Supervised Representation Learning](https://mlsanity.com/paper/2304.11718/) (2023)
- [Guided Masked Self-Distillation Modeling for Distributed Multimedia Sensor Event Analysis](https://mlsanity.com/paper/2404.08264/) (2024)
- [Self-Supervised Learning for Place Representation Generalization across Appearance Changes](https://mlsanity.com/paper/2303.02370/) (2023)
- [Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion](https://mlsanity.com/paper/2402.16305/) (2024)
- [Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning](https://mlsanity.com/paper/2304.05600/) (2023)
- [PreViTS: Contrastive Pretraining with Video Tracking Supervision](https://mlsanity.com/paper/2112.00804/) (2021)
- [On Negative Sampling for Audio-Visual Contrastive Learning from Movies](https://mlsanity.com/paper/2205.00073/) (2022)
- [3D-Augmented Contrastive Knowledge Distillation for Image-based Object Pose Estimation](https://mlsanity.com/paper/2206.02531/) (2022)
- [Optimizing transformations for contrastive learning in a differentiable framework](https://mlsanity.com/paper/2207.13367/) (2022)
- [DETECLAP: Enhancing Audio-Visual Representation Learning with Object Information](https://mlsanity.com/paper/2409.11729/) (2024)
- [Less than Few: Self-Shot Video Instance Segmentation](https://mlsanity.com/paper/2204.08874/) (2022)

## Abstract (arXiv)

In the image domain, excellent representations can be learned by inducing invariance to content-preserving transformations via noise contrastive learning. In this paper, we generalize contrastive learning to a wider set of transformations, and their compositions, for which either invariance or distinctiveness is sought. We show that it is not immediately obvious how existing methods such as SimCLR can be extended to do so. Instead, we introduce a number of formal requirements that all contrastive formulations must satisfy, and propose a practical construction which satisfies these requirements. In order to maximise the reach of this analysis, we express all components of noise contrastive formulations as the choice of certain generalized transformations of the data (GDTs), including data sampling. We then consider videos as an example of data in which a large variety of transformations are applicable, accounting for the extra modalities -- for which we analyze audio and text -- and the dimension of time. We find that being invariant to certain transformations and distinctive to others is critical to learning effective video representations, improving the state-of-the-art for multiple benchmarks by a large margin, and even surpassing supervised pretraining.