# AdaDim: Dimensionality Adaptation for SSL Representational Dynamics

arXiv: https://arxiv.org/abs/2505.12576 · HTML: https://mlsanity.com/paper/2505.12576/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 56 references that are in the map.
- First posted on arXiv 18 May 2025.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [Understanding self-supervised Learning Dynamics without Contrastive Pairs](https://mlsanity.com/paper/2102.06810/) (2021)
- [Learning Robust Representations via Multi-View Information Bottleneck](https://mlsanity.com/paper/2002.07017/) (2020)
- [RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank](https://mlsanity.com/paper/2210.02885/) (2022)
- [Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning](https://mlsanity.com/paper/2108.01775/) (2021)
- [On the duality between contrastive and non-contrastive self-supervised learning](https://mlsanity.com/paper/2206.02574/) (2022)
- [ReSSL: Relational Self-Supervised Learning with Weak Augmentation](https://mlsanity.com/paper/2107.09282/) (2021)
- [Critical Learning Periods in Deep Neural Networks](https://mlsanity.com/paper/1711.08856/) (2017)
- [On the Stepwise Nature of Self-Supervised Learning](https://mlsanity.com/paper/2303.15438/) (2023)
- [Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism](https://mlsanity.com/paper/2303.02387/) (2023)

## Built on by (most cited first, 4 of 4)

- [Countering Multi-modal Representation Collapse through Rank-targeted Fusion](https://mlsanity.com/paper/2511.06450/) (2025)
- [Information Router for Mitigating Modality Dominance in Vision-Language Models](https://mlsanity.com/paper/2604.16264/) (2026)
- [MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization](https://mlsanity.com/paper/2605.01967/) (2026)
- [GeoPID: Decomposing and Steering Visual Information in Vision-Language Models](https://mlsanity.com/paper/2610.08401/) (2026)

## Abstract (arXiv)

A key factor in effective Self-Supervised learning (SSL) is preventing dimensional collapse, where higher-dimensional representation spaces (R) span a lower-dimensional subspace. Therefore, SSL optimization strategies involve guiding a model to produce R with a higher dimensionality (H(R)) through objectives that encourage decorrelation of features or sample uniformity in R. A higher H(R) indicates that R has greater feature diversity which is useful for generalization to downstream tasks. Alongside dimensionality optimization, SSL algorithms also utilize a projection head that maps R into an embedding space Z. Recent work has characterized the projection head as a filter of noisy or irrelevant features from the SSL objective by reducing the mutual information I(R;Z). Therefore, the current literature's view is that a good SSL representation space should have a high H(R) and a low I(R;Z). However, this view of SSL is lacking in terms of an understanding of the underlying training dynamics that influences the relationship between both terms. Our analysis shows that the best performing SSL models do not have the highest H(R) nor the lowest I(R;Z), but effectively arrive at a balance between both. To take advantage of this analysis, we introduce AdaDim, a training strategy that leverages SSL training dynamics by adaptively balancing between increasing H(R) through feature decorrelation and sample uniformity as well as gradual regularization of I(R;Z) as training progresses. We show performance improvements of up to 3% over common SSL baselines despite our method not utilizing expensive techniques such as queues, clustering, predictor networks, or student-teacher architectures.