# I-Con: A Unifying Framework for Representation Learning

arXiv: https://arxiv.org/abs/2504.16929 · HTML: https://mlsanity.com/paper/2504.16929/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Contrastive Clustering](https://mlsanity.com/paper/2009.09687/) (2020)
- [Unified Contrastive Learning in Image-Text-Label Space](https://mlsanity.com/paper/2204.03610/) (2022)
- [Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods](https://mlsanity.com/paper/2205.11508/) (2022)
- [Exploring the Limits of Deep Image Clustering using Pretrained Models](https://mlsanity.com/paper/2303.17896/) (2023)
- [Learning Visual Representations via Language-Guided Sampling](https://mlsanity.com/paper/2302.12248/) (2023)
- [Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embedding](https://mlsanity.com/paper/2205.14814/) (2022)
- [Unsupervised visualization of image datasets using contrastive learning](https://mlsanity.com/paper/2210.09879/) (2022)
- [mathbbX-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs](https://mlsanity.com/paper/2407.18134/) (2024)
- [Harmonic Loss Trains Interpretable AI Models](https://mlsanity.com/paper/2502.01628/) (2025)
- [Word-Graph2vec: An efficient word embedding approach on word co-occurrence graph using random walk technique](https://mlsanity.com/paper/2301.04312/) (2023)

## Built on by (most cited first, 16 of 16)

- [Nested Learning: The Illusion of Deep Learning Architectures](https://mlsanity.com/paper/2512.24695/) (2025)
- [Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules](https://mlsanity.com/paper/2510.26997/) (2025)
- [Understanding Self-Supervised Learning via Latent Distribution Matching](https://mlsanity.com/paper/2605.03517/) (2026)
- [Back To The Drawing Board: Rethinking Scene-Level Sketch-Based Image Retrieval](https://mlsanity.com/paper/2509.06566/) (2025)
- [Recursive KL Divergence Optimization: A Dynamic Framework for Representation Learning](https://mlsanity.com/paper/2504.21707/) (2025)
- [Supervised Graph Contrastive Learning for Gene Regulatory Networks](https://mlsanity.com/paper/2505.17786/) (2025)
- [Enhancing VICReg: Random-Walk Pairing for Improved Generalization and Better Global Semantics Capturing](https://mlsanity.com/paper/2506.18104/) (2025)
- [Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder](https://mlsanity.com/paper/2506.20083/) (2025)
- [Beyond the Linear Separability Ceiling: Aligning Representations in VLMs](https://mlsanity.com/paper/2507.07574/) (2025)
- [The Impact of Semantic Pairs on Self-Supervised Representation Learning](https://mlsanity.com/paper/2510.08722/) (2025)
- [Network of Theseus (like the ship)](https://mlsanity.com/paper/2512.04198/) (2025)
- [Learning Minimal Representations of Fermionic Ground States](https://mlsanity.com/paper/2512.11767/) (2025)
- [Group Contrastive Learning for Weakly Paired Multimodal Data](https://mlsanity.com/paper/2602.04021/) (2026)
- [EGA: Adapting Frozen Encoders for Vector Search with Bounded Out-of-Distribution Degradation](https://mlsanity.com/paper/2605.05674/) (2026)
- [When Softmax Fails at the Top: Extreme Value Corrections for InfoNCE](https://mlsanity.com/paper/2606.00262/) (2026)
- [To Align or Not To Align: Check Your COMPASS Before You Train](https://mlsanity.com/paper/2610.18638/) (2026)

## Abstract (arXiv)

As the field of representation learning grows, there has been a proliferation of different loss functions to solve different classes of problems. We introduce a single information-theoretic equation that generalizes a large collection of modern loss functions in machine learning. In particular, we introduce a framework that shows that several broad classes of machine learning methods are precisely minimizing an integrated KL divergence between two conditional distributions: the supervisory and learned representations. This viewpoint exposes a hidden information geometry underlying clustering, spectral methods, dimensionality reduction, contrastive learning, and supervised learning. This framework enables the development of new loss functions by combining successful techniques from across the literature. We not only present a wide array of proofs, connecting over 23 different approaches, but we also leverage these theoretical results to create state-of-the-art unsupervised image classifiers that achieve a +8% improvement over the prior state-of-the-art on unsupervised classification on ImageNet-1K. We also demonstrate that I-Con can be used to derive principled debiasing methods which improve contrastive representation learners.