# Manifold Contrastive Learning with Variational Lie Group Operators

arXiv: https://arxiv.org/abs/2306.13544 · HTML: https://mlsanity.com/paper/2306.13544/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 77 references that are in the map.
- First posted on arXiv 23 Jun 2023.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [The Intrinsic Dimension of Images and Its Impact on Learning](https://mlsanity.com/paper/2104.08894/) (2021)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [Interpolation Consistency Training for Semi-Supervised Learning](https://mlsanity.com/paper/1903.03825/) (2019)
- [Excessive Invariance Causes Adversarial Vulnerability](https://mlsanity.com/paper/1811.00401/) (2018)
- [FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning](https://mlsanity.com/paper/2007.08505/) (2020)
- [Accurate and Diverse Sampling of Sequences based on a "Best of Many" Sample Objective](https://mlsanity.com/paper/1806.07772/) (2018)
- [Variational Autoencoder with Learned Latent Structure](https://mlsanity.com/paper/2006.10597/) (2020)
- [The Sparse Manifold Transform](https://mlsanity.com/paper/1806.08887/) (2018)
- [LieGG: Studying Learned Lie Group Generators](https://mlsanity.com/paper/2210.04345/) (2022)
- [Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations](https://mlsanity.com/paper/2303.03307/) (2023)
- [Representing Closed Transformation Paths in Encoded Network Latent Space](https://mlsanity.com/paper/1912.02644/) (2019)
- [Toward a Geometrical Understanding of Self-supervised Contrastive Learning](https://mlsanity.com/paper/2205.06926/) (2022)
- [Addressing the Topological Defects of Disentanglement via Distributed Operators](https://mlsanity.com/paper/2102.05623/) (2021)

## Built on by (most cited first, 1 of 1)

- [MANGO: Learning Disentangled Image Transformation Manifolds with Grouped Operators](https://mlsanity.com/paper/2409.09542/) (2024)

## Abstract (arXiv)

Self-supervised learning of deep neural networks has become a prevalent paradigm for learning representations that transfer to a variety of downstream tasks. Similar to proposed models of the ventral stream of biological vision, it is observed that these networks lead to a separation of category manifolds in the representations of the penultimate layer. Although this observation matches the manifold hypothesis of representation learning, current self-supervised approaches are limited in their ability to explicitly model this manifold. Indeed, current approaches often only apply augmentations from a pre-specified set of "positive pairs" during learning. In this work, we propose a contrastive learning approach that directly models the latent manifold using Lie group operators parameterized by coefficients with a sparsity-promoting prior. A variational distribution over these coefficients provides a generative model of the manifold, with samples which provide feature augmentations applicable both during contrastive training and downstream tasks. Additionally, learned coefficient distributions provide a quantification of which transformations are most likely at each point on the manifold while preserving identity. We demonstrate benefits in self-supervised benchmarks for image datasets, as well as a downstream semi-supervised task. In the former case, we demonstrate that the proposed methods can effectively apply manifold feature augmentations and improve learning both with and without a projection head. In the latter case, we demonstrate that feature augmentations sampled from learned Lie group operators can improve classification performance when using few labels.