# Beyond Pairwise Correlations: Higher-Order Redundancies in Self-Supervised Representation Learning

arXiv: https://arxiv.org/abs/2412.01926 · HTML: https://mlsanity.com/paper/2412.01926/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 26 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Understanding and Improving the Role of Projection Head in Self-Supervised Learning](https://mlsanity.com/paper/2212.11491/) (2022)
- [A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning](https://mlsanity.com/paper/2104.13712/) (2021)
- [Computer Vision Self-supervised Learning Methods on Time Series](https://mlsanity.com/paper/2109.00783/) (2021)

## Built on by (most cited first, 4 of 4)

- [Adversarial Dependence Minimization](https://mlsanity.com/paper/2502.03227/) (2025)
- [Redundancy as a Structural Information Principle for Learning and Generalization](https://mlsanity.com/paper/2510.10938/) (2025)
- [Beyond Augmentation: Leveraging Inter-Instance Relation in Self-Supervised Representation Learning](https://mlsanity.com/paper/2510.22322/) (2025)
- [Information-driven Fusion of Pathology Foundation Models for Enhanced Disease Characterization](https://mlsanity.com/paper/2512.11104/) (2025)

## Abstract (arXiv)

Several self-supervised learning (SSL) approaches have shown that redundancy reduction in the feature embedding space is an effective tool for representation learning. However, these methods consider a narrow notion of redundancy, focusing on pairwise correlations between features. To address this limitation, we formalize the notion of embedding space redundancy and introduce redundancy measures that capture more complex, higher-order dependencies. We mathematically analyze the relationships between these metrics, and empirically measure these redundancies in the embedding spaces of common SSL methods. Based on our findings, we propose Self Supervised Learning with Predictability Minimization (SSLPM) as a method for reducing redundancy in the embedding space. SSLPM combines an encoder network with a predictor engaging in a competitive game of reducing and exploiting dependencies respectively. We demonstrate that SSLPM is competitive with state-of-the-art methods and find that the best performing SSL methods exhibit low embedding space redundancy, suggesting that even methods without explicit redundancy reduction mechanisms perform redundancy reduction implicitly.