# Beyond I-Con: Exploring New Dimension of Distance Measures in Representation Learning

arXiv: https://arxiv.org/abs/2509.04734 · HTML: https://mlsanity.com/paper/2509.04734/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 12 references that are in the map.
- First posted on arXiv 5 Sep 2025.

## Builds on (most cited first, 6 of 6)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Exploring the Limits of Deep Image Clustering using Pretrained Models](https://mlsanity.com/paper/2303.17896/) (2023)
- [Optimization Issues in KL-Constrained Approximate Policy Iteration](https://mlsanity.com/paper/2102.06234/) (2021)

## Built on by (most cited first, 1 of 1)

- [LESS Is More: Mutual-Stability Sampling for Diffusion Language Models](https://mlsanity.com/paper/2606.16908/) (2026)

## Abstract (arXiv)

The Information Contrastive (I-Con) framework revealed that over 23 representation learning methods implicitly minimize KL divergence between data and learned distributions that encode similarities between data points. However, a KL-based loss may be misaligned with the true objective, and properties of KL divergence such as asymmetry and unboundedness may create optimization challenges. We present Beyond I-Con, a framework that enables systematic discovery of novel loss functions by exploring alternative statistical divergences. Key findings: (1) on unsupervised clustering of DINO-ViT embeddings, we achieve state-of-the-art results by modifying the PMI algorithm to use total variation (TV) distance; (2) supervised contrastive learning with Euclidean distance as the feature space metric is improved by replacing the standard loss function with Jenson-Shannon divergence (JSD); (3) on dimensionality reduction, we achieve superior qualitative results and better performance on downstream tasks than SNE by replacing KL with a bounded f-divergence. Our results highlight the importance of considering divergence choices in representation learning optimization.