# ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representations

arXiv: https://arxiv.org/abs/2303.01092 · HTML: https://mlsanity.com/paper/2303.01092/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 57 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 30 of 42)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Domain Generalization using Causal Matching](https://mlsanity.com/paper/2006.07500/) (2020)
- [How Well Do Self-Supervised Models Transfer?](https://mlsanity.com/paper/2011.13377/) (2020)
- [Self-supervised Pretraining of Visual Features in the Wild](https://mlsanity.com/paper/2103.01988/) (2021)
- [Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach](https://mlsanity.com/paper/1610.03425/) (2016)
- [Robust Pre-Training by Adversarial Contrastive Learning](https://mlsanity.com/paper/2010.13337/) (2020)
- [Adversarial Self-Supervised Contrastive Learning](https://mlsanity.com/paper/2006.07589/) (2020)
- [Self-supervised Learning is More Robust to Dataset Imbalance](https://mlsanity.com/paper/2110.05025/) (2021)
- [Generative Models as a Data Source for Multiview Representation Learning](https://mlsanity.com/paper/2106.05258/) (2021)
- [Towards the Generalization of Contrastive Self-Supervised Learning](https://mlsanity.com/paper/2111.00743/) (2021)

## Built on by (most cited first, 10 of 10)

- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [GraphCLIP: Enhancing Transferability in Graph Foundation Models for Text-Attributed Graphs](https://mlsanity.com/paper/2410.10329/) (2024)
- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [Unsupervised Representation Learning for Time Series: A Review](https://mlsanity.com/paper/2308.01578/) (2023)
- [Adversarial Representation Engineering: A General Model Editing Framework for Large Language Models](https://mlsanity.com/paper/2404.13752/) (2024)
- [Tri-Level Navigator: LLM-Empowered Tri-Level Learning for Time Series OOD Generalization](https://mlsanity.com/paper/2410.07018/) (2024)
- [Task-Oriented Koopman-Based Control with Contrastive Encoder](https://mlsanity.com/paper/2309.16077/) (2023)
- [MARIO: Model Agnostic Recipe for Improving OOD Generalization of Graph Contrastive Learning](https://mlsanity.com/paper/2307.13055/) (2023)
- [Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining](https://mlsanity.com/paper/2407.00935/) (2024)
- [Towards Understanding the Mechanism of Contrastive Learning via Similarity Structure: A Theoretical Analysis](https://mlsanity.com/paper/2304.00395/) (2023)

## Abstract (arXiv)

Self-Supervised Learning (SSL) is a paradigm that leverages unlabeled data for model training. Empirical studies show that SSL can achieve promising performance in distribution shift scenarios, where the downstream and training distributions differ. However, the theoretical understanding of its transferability remains limited. In this paper, we develop a theoretical framework to analyze the transferability of self-supervised contrastive learning, by investigating the impact of data augmentation on it. Our results reveal that the downstream performance of contrastive learning depends largely on the choice of data augmentation. Moreover, we show that contrastive learning fails to learn domain-invariant features, which limits its transferability. Based on these theoretical insights, we propose a novel method called Augmentation-robust Contrastive Learning (ArCL), which guarantees to learn domain-invariant features and can be easily integrated with existing contrastive learning algorithms. We conduct experiments on several datasets and show that ArCL significantly improves the transferability of contrastive learning.