# EquiMod: An Equivariance Module to Improve Self-Supervised Learning

arXiv: https://arxiv.org/abs/2211.01244 · HTML: https://mlsanity.com/paper/2211.01244/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 2 Nov 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [What Should Not Be Contrastive in Contrastive Learning](https://mlsanity.com/paper/2008.05659/) (2020)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [Can contrastive learning avoid shortcut solutions?](https://mlsanity.com/paper/2106.11230/) (2021)
- [Equivariant Contrastive Learning](https://mlsanity.com/paper/2111.00899/) (2021)
- [Learning image representations tied to ego-motion](https://mlsanity.com/paper/1505.02206/) (2015)
- [Understand and Improve Contrastive Learning Methods for Visual Representation: A Review](https://mlsanity.com/paper/2106.03259/) (2021)

## Built on by (most cited first, 21 of 21)

- [CheXWorld: Exploring Image World Modeling for Radiograph Representation Learning](https://mlsanity.com/paper/2504.13820/) (2025)
- [EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning](https://mlsanity.com/paper/2403.09502/) (2024)
- [Self-supervised visual learning from interactions with objects](https://mlsanity.com/paper/2407.06704/) (2024)
- [seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models](https://mlsanity.com/paper/2505.03176/) (2025)
- [Towards a Unified Representation Evaluation Framework Beyond Downstream Tasks](https://mlsanity.com/paper/2505.06224/) (2025)
- [Contrastive Learning with Consistent Representations](https://mlsanity.com/paper/2302.01541/) (2023)
- [Improving Self-supervised Molecular Representation Learning using Persistent Homology](https://mlsanity.com/paper/2311.17327/) (2023)
- [3DTINC: Time-Equivariant Non-Contrastive Learning for Predicting Disease Progression from Longitudinal OCTs](https://mlsanity.com/paper/2312.16980/) (2023)
- [PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objective](https://mlsanity.com/paper/2508.01488/) (2025)
- [Capsule Network Projectors are Equivariant and Invariant Learners](https://mlsanity.com/paper/2405.14386/) (2024)
- [Equivariance by Contrast: Identifiable Equivariant Embeddings from Unlabeled Finite Group Actions](https://mlsanity.com/paper/2510.21706/) (2025)
- [Learning Temporally Equivariance for Degenerative Disease Progression in OCT by Predicting Future Representations](https://mlsanity.com/paper/2405.09404/) (2024)
- [Point-PNG: Conditional Pseudo-Negatives Generation for Point Cloud Pre-Training](https://mlsanity.com/paper/2409.15832/) (2024)
- [Self-Supervised Learning Based on Transformed Image Reconstruction for Equivariance-Coherent Feature Representation](https://mlsanity.com/paper/2503.18753/) (2025)
- [OWT: A Foundational Organ-Wise Tokenization Framework for Medical Imaging](https://mlsanity.com/paper/2505.04899/) (2025)
- [Self-Supervised Learning from Structural Invariance](https://mlsanity.com/paper/2602.02381/) (2026)
- [Soft Equivariance Regularization for Invariant Self-Supervised Learning](https://mlsanity.com/paper/2603.06693/) (2026)
- [Semantic Equivariant Mixup](https://mlsanity.com/paper/2308.06451/) (2023)
- [Mirror-Fusion Attention for Reflection-Aware Self-Supervised Representation Learning](https://mlsanity.com/paper/2607.00850/) (2026)
- [Reformulation-Contrastive Learning for Mixed Integer Programs](https://mlsanity.com/paper/2610.00730/) (2026)
- [Variance Fingerprints: Component-Wise Identification of Content Factors by Augmentation Design](https://mlsanity.com/paper/2610.14260/) (2026)

## Abstract (arXiv)

Self-supervised visual representation methods are closing the gap with supervised learning performance. These methods rely on maximizing the similarity between embeddings of related synthetic inputs created through data augmentations. This can be seen as a task that encourages embeddings to leave out factors modified by these augmentations, i.e. to be invariant to them. However, this only considers one side of the trade-off in the choice of the augmentations: they need to strongly modify the images to avoid simple solution shortcut learning (e.g. using only color histograms), but on the other hand, augmentations-related information may be lacking in the representations for some downstream tasks (e.g. color is important for birds and flower classification). Few recent works proposed to mitigate the problem of using only an invariance task by exploring some form of equivariance to augmentations. This has been performed by learning additional embeddings space(s), where some augmentation(s) cause embeddings to differ, yet in a non-controlled way. In this work, we introduce EquiMod a generic equivariance module that structures the learned latent space, in the sense that our module learns to predict the displacement in the embedding space caused by the augmentations. We show that applying that module to state-of-the-art invariance models, such as SimCLR and BYOL, increases the performances on CIFAR10 and ImageNet datasets. Moreover, while our model could collapse to a trivial equivariance, i.e. invariance, we observe that it instead automatically learns to keep some augmentations-related information beneficial to the representations.