# Residual Relaxation for Multi-view Representation Learning

arXiv: https://arxiv.org/abs/2110.15348 · HTML: https://mlsanity.com/paper/2110.15348/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [What Should Not Be Contrastive in Contrastive Learning](https://mlsanity.com/paper/2008.05659/) (2020)
- [A Survey of Multi-View Representation Learning](https://mlsanity.com/paper/1610.01206/) (2016)

## Built on by (most cited first, 30 of 30)

- [Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap](https://mlsanity.com/paper/2203.13457/) (2022)
- [How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders](https://mlsanity.com/paper/2210.08344/) (2022)
- [Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism](https://mlsanity.com/paper/2303.02387/) (2023)
- [On the Generalization of Multi-modal Contrastive Learning](https://mlsanity.com/paper/2306.04272/) (2023)
- [A Canonicalization Perspective on Invariant and Equivariant Learning](https://mlsanity.com/paper/2405.18378/) (2024)
- [Do Generated Data Always Help Contrastive Learning?](https://mlsanity.com/paper/2403.12448/) (2024)
- [A Message Passing Perspective on Learning Dynamics of Contrastive Learning](https://mlsanity.com/paper/2303.04435/) (2023)
- [Rethinking the Effect of Data Augmentation in Adversarial Contrastive Learning](https://mlsanity.com/paper/2303.01289/) (2023)
- [Non-negative Contrastive Learning](https://mlsanity.com/paper/2403.12459/) (2024)
- [Rethinking Weak Supervision in Helping Contrastive Learning](https://mlsanity.com/paper/2306.04160/) (2023)
- [Adversarial Examples Are Not Real Features](https://mlsanity.com/paper/2310.18936/) (2023)
- [A Novel Approach for Effective Multi-View Clustering with Information-Theoretic Perspective](https://mlsanity.com/paper/2309.13989/) (2023)
- [Identifiable Contrastive Learning with Automatic Feature Importance Discovery](https://mlsanity.com/paper/2310.18904/) (2023)
- [Projection Head is Secretly an Information Bottleneck](https://mlsanity.com/paper/2503.00507/) (2025)
- [Decoupled Self-supervised Learning for Non-Homophilous Graphs](https://mlsanity.com/paper/2206.03601/) (2022)
- [Architecture Matters: Uncovering Implicit Mechanisms in Graph Contrastive Learning](https://mlsanity.com/paper/2311.02687/) (2023)
- [On the Role of Discrete Tokenization in Visual Representation Learning](https://mlsanity.com/paper/2407.09087/) (2024)
- [Understanding the Role of Equivariance in Self-supervised Learning](https://mlsanity.com/paper/2411.06508/) (2024)
- [Feature Disentanglement Learning with Switching and Aggregation for Video-based Person Re-Identification](https://mlsanity.com/paper/2212.09498/) (2022)
- [Towards Unsupervised Domain Bridging via Image Degradation in Semantic Segmentation](https://mlsanity.com/paper/2412.10339/) (2024)
- [An Augmentation Overlap Theory of Contrastive Learning](https://mlsanity.com/paper/2511.03114/) (2025)
- [Self-Supervised Learning for Place Representation Generalization across Appearance Changes](https://mlsanity.com/paper/2303.02370/) (2023)
- [Rethinking Rotation in Self-Supervised Contrastive Learning: Adaptive Positive or Negative Data Augmentation](https://mlsanity.com/paper/2210.12681/) (2022)
- [Homomorphic Self-Supervised Learning](https://mlsanity.com/paper/2211.08282/) (2022)
- [Mitigating Forgetting in Online Continual Learning via Contrasting Semantically Distinct Augmentations](https://mlsanity.com/paper/2211.05347/) (2022)
- [An Inclusive Theoretical Framework of Robust Supervised Contrastive Loss against Label Noise](https://mlsanity.com/paper/2501.01130/) (2025)
- [Difficult Examples Hurt Unsupervised Contrastive Learning: A Theoretical Perspective](https://mlsanity.com/paper/2501.01317/) (2025)
- [CIPER: Combining Invariant and Equivariant Representations Using Contrastive and Predictive Learning](https://mlsanity.com/paper/2302.02330/) (2023)
- [Adaptive Augmentation-Aware Latent Learning for Robust LiDAR Semantic Segmentation](https://mlsanity.com/paper/2603.01074/) (2026)
- [A Theoretical Framework for Masked Pretraining (MPT)](https://mlsanity.com/paper/2609.06460/) (2026)

## Abstract (arXiv)

Multi-view methods learn representations by aligning multiple views of the same image and their performance largely depends on the choice of data augmentation. In this paper, we notice that some other useful augmentations, such as image rotation, are harmful for multi-view methods because they cause a semantic shift that is too large to be aligned well. This observation motivates us to relax the exact alignment objective to better cultivate stronger augmentations. Taking image rotation as a case study, we develop a generic approach, Pretext-aware Residual Relaxation (Prelax), that relaxes the exact alignment by allowing an adaptive residual vector between different views and encoding the semantic shift through pretext-aware learning. Extensive experiments on different backbones show that our method can not only improve multi-view methods with existing augmentations, but also benefit from stronger image augmentations like rotation.