# Structuring Representation Geometry with Rotationally Equivariant Contrastive Learning

arXiv: https://arxiv.org/abs/2306.13924 · HTML: https://mlsanity.com/paper/2306.13924/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 24 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks](https://mlsanity.com/paper/2009.11848/) (2020)
- [Deep Bidirectional Language-Knowledge Graph Pretraining](https://mlsanity.com/paper/2210.09338/) (2022)
- [Equivariant Contrastive Learning](https://mlsanity.com/paper/2111.00899/) (2021)
- [Hyperbolic Contrastive Learning for Visual Representations beyond Objects](https://mlsanity.com/paper/2212.00653/) (2022)
- [On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligence](https://mlsanity.com/paper/2207.04630/) (2022)
- [Self-supervised learning of Split Invariant Equivariant representations](https://mlsanity.com/paper/2302.10283/) (2023)
- [Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism](https://mlsanity.com/paper/2303.02387/) (2023)
- [Learning Group Structure and Disentangled Representations of Dynamical Environments](https://mlsanity.com/paper/2002.06991/) (2020)
- [Steerable Equivariant Representation Learning](https://mlsanity.com/paper/2302.11349/) (2023)

## Built on by (most cited first, 25 of 25)

- [Steerers: A framework for rotation equivariant keypoint descriptors](https://mlsanity.com/paper/2312.02152/) (2023)
- [Approximately Equivariant Graph Networks](https://mlsanity.com/paper/2308.10436/) (2023)
- [OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence](https://mlsanity.com/paper/2602.08683/) (2026)
- [Flopping for FLOPs: Leveraging equivariance for computational efficiency](https://mlsanity.com/paper/2502.05169/) (2025)
- [Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses](https://mlsanity.com/paper/2405.18045/) (2024)
- [Understanding the Role of Equivariance in Self-supervised Learning](https://mlsanity.com/paper/2411.06508/) (2024)
- [seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models](https://mlsanity.com/paper/2505.03176/) (2025)
- [Affine steerers for structured keypoint description](https://mlsanity.com/paper/2408.14186/) (2024)
- [In-Context Symmetries: Self-Supervised Learning through Contextual World Models](https://mlsanity.com/paper/2405.18193/) (2024)
- [Learning Color Equivariant Representations](https://mlsanity.com/paper/2406.09588/) (2024)
- [Global Minimizers of Sigmoid Contrastive Loss](https://mlsanity.com/paper/2509.18552/) (2025)
- [Incorporating Arbitrary Matrix Group Equivariance into KANs](https://mlsanity.com/paper/2410.00435/) (2024)
- [Tensor learning with orthogonal, Lorentz, and symplectic symmetries](https://mlsanity.com/paper/2406.01552/) (2024)
- [Equivariance by Contrast: Identifiable Equivariant Embeddings from Unlabeled Finite Group Actions](https://mlsanity.com/paper/2510.21706/) (2025)
- [TriMotion: Modality-Agnostic Camera Control for Video Generation](https://mlsanity.com/paper/2606.20774/) (2026)
- [Contrastive Learning Via Equivariant Representation](https://mlsanity.com/paper/2406.00262/) (2024)
- [Point-PNG: Conditional Pseudo-Negatives Generation for Point Cloud Pre-Training](https://mlsanity.com/paper/2409.15832/) (2024)
- [Self-Supervised Learning Based on Transformed Image Reconstruction for Equivariance-Coherent Feature Representation](https://mlsanity.com/paper/2503.18753/) (2025)
- [Soft Equivariance Regularization for Invariant Self-Supervised Learning](https://mlsanity.com/paper/2603.06693/) (2026)
- [From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs](https://mlsanity.com/paper/2603.26323/) (2026)
- [EquiCaps: Predictor-Free Pose-Aware Pre-Trained Capsule Networks](https://mlsanity.com/paper/2506.09895/) (2025)
- [Eigenfunction Extraction for Ordered Representation Learning](https://mlsanity.com/paper/2510.24672/) (2025)
- [Soft Task-Aware Routing of Experts for Equivariant Representation Learning](https://mlsanity.com/paper/2510.27222/) (2025)
- [LEPA: Learning Geometric Equivariance in Satellite Remote Sensing Data with a Predictive Architecture](https://mlsanity.com/paper/2603.07246/) (2026)
- [Beyond Task Success: A Theory of Representation Reuse](https://mlsanity.com/paper/2610.18375/) (2026)

## Abstract (arXiv)

Self-supervised learning converts raw perceptual data such as images to a compact space where simple Euclidean distances measure meaningful variations in data. In this paper, we extend this formulation by adding additional geometric structure to the embedding space by enforcing transformations of input space to correspond to simple (i.e., linear) transformations of embedding space. Specifically, in the contrastive learning setting, we introduce an equivariance objective and theoretically prove that its minima forces augmentations on input space to correspond to rotations on the spherical embedding space. We show that merely combining our equivariant loss with a non-collapse term results in non-trivial representations, without requiring invariance to data augmentations. Optimal performance is achieved by also encouraging approximate invariance, where input augmentations correspond to small rotations. Our method, CARE: Contrastive Augmentation-induced Rotational Equivariance, leads to improved performance on downstream tasks, and ensures sensitivity in embedding space to important variations in data (e.g., color) that standard contrastive methods do not achieve. Code is available at https://github.com/Sharut/CARE.