# Rethinking Self-Supervised Learning Within the Framework of Partial Information Decomposition

arXiv: https://arxiv.org/abs/2412.02121 · HTML: https://mlsanity.com/paper/2412.02121/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 3 Dec 2024.

## Builds on (most cited first, 20 of 20)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [Scaling and Benchmarking Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1905.01235/) (2019)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning](https://mlsanity.com/paper/2108.01775/) (2021)
- [Tighter Variational Bounds are Not Necessarily Better](https://mlsanity.com/paper/1802.04537/) (2018)
- [Learning Where to Learn in Cross-View Self-Supervised Learning](https://mlsanity.com/paper/2203.14898/) (2022)
- [Decomposed Mutual Information Estimation for Contrastive Representation Learning](https://mlsanity.com/paper/2106.13401/) (2021)
- [Bits and Pieces: Understanding Information Decomposition from Part-whole Relationships and Formal Logic](https://mlsanity.com/paper/2008.09535/) (2020)
- [Directional Self-supervised Learning for Heavy Image Augmentations](https://mlsanity.com/paper/2110.13555/) (2021)
- [More Synergy, Less Redundancy: Exploiting Joint Mutual Information for Self-Supervised Learning](https://mlsanity.com/paper/2307.00651/) (2023)
- [Deep Active Ensemble Sampling For Image Classification](https://mlsanity.com/paper/2210.05770/) (2022)
- [FUSSL: Fuzzy Uncertain Self Supervised Learning](https://mlsanity.com/paper/2210.15818/) (2022)

## Abstract (arXiv)

Self Supervised learning (SSL) has demonstrated its effectiveness in feature learning from unlabeled data. Regarding this success, there have been some arguments on the role that mutual information plays within the SSL framework. Some works argued for increasing mutual information between representation of augmented views. Others suggest decreasing mutual information between them, while increasing task-relevant information. We ponder upon this debate and propose to revisit the core idea of SSL within the framework of partial information decomposition (PID). Thus, with SSL under PID we propose to replace traditional mutual information with the more general concept of joint mutual information to resolve the argument. Our investigation on instantiation of SSL within the PID framework leads to upgrading the existing pipelines by considering the components of the PID in the SSL models for improved representation learning. Accordingly we propose a general pipeline that can be applied to improve existing baselines. Our pipeline focuses on extracting the unique information component under the PID to build upon lower level supervision for generic feature learning and on developing higher-level supervisory signals for task-related feature learning. In essence, this could be interpreted as a joint utilization of local and global clustering. Experiments on four baselines and four datasets show the effectiveness and generality of our approach in improving existing SSL frameworks.