# Connecting Jensen-Shannon and Kullback-Leibler Divergences: A New Bound for Representation Learning

arXiv: https://arxiv.org/abs/2510.20644 · HTML: https://mlsanity.com/paper/2510.20644/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 23 Oct 2025.

## Builds on (most cited first, 17 of 17)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Nonlinear Information Bottleneck](https://mlsanity.com/paper/1705.02436/) (2017)
- [Beyond Normal: On the Evaluation of Mutual Information Estimators](https://mlsanity.com/paper/2306.11078/) (2023)
- [Disentangled Information Bottleneck](https://mlsanity.com/paper/2012.07372/) (2020)
- [Neural Methods for Point-wise Dependency Estimation](https://mlsanity.com/paper/2006.05553/) (2020)
- [Mutual Information Estimation via f-Divergence and Data Derangements](https://mlsanity.com/paper/2305.20025/) (2023)
- [The Convex Information Bottleneck Lagrangian](https://mlsanity.com/paper/1911.11000/) (2019)
- [DEMI: Discriminative Estimator of Mutual Information](https://mlsanity.com/paper/2010.01766/) (2020)
- [Properties of f-divergences and f-GAN training](https://mlsanity.com/paper/2009.00757/) (2020)

## Built on by (most cited first, 4 of 4)

- [Not all Jensen-Shannon Divergence Estimators are Equal](https://mlsanity.com/paper/2606.16411/) (2026)
- [SynPre-FL: Synthetic data-driven pretraining integrated Federated Learning training framework](https://mlsanity.com/paper/2607.19524/) (2026)
- [CPDA: Class-Conditional Path Distribution Alignment for Unsupervised Time-Series Domain Adaptation](https://mlsanity.com/paper/2608.09193/) (2026)
- [On the Trade-off Between Information Loss and Generalization in Sparse Attention](https://mlsanity.com/paper/2610.04424/) (2026)

## Abstract (arXiv)

Mutual Information (MI) is a fundamental measure of statistical dependence widely used in representation learning. While direct optimization of MI via its definition as a Kullback-Leibler divergence (KLD) is often intractable, many recent methods have instead maximized alternative dependence measures, most notably, the Jensen-Shannon divergence (JSD) between joint and product of marginal distributions via discriminative losses. However, the connection between these surrogate objectives and MI remains poorly understood. In this work, we bridge this gap by deriving a new, tight, and tractable lower bound on KLD as a function of JSD in the general case. By specializing this bound to joint and marginal distributions, we demonstrate that maximizing the JSD-based information increases a guaranteed lower bound on mutual information. Furthermore, we revisit the practical implementation of JSD-based objectives and observe that minimizing the cross-entropy loss of a binary classifier trained to distinguish joint from marginal pairs recovers a known variational lower bound on the JSD. Extensive experiments demonstrate that our lower bound is tight when applied to MI estimation. We compared our lower bound to state-of-the-art neural estimators of variational lower bound across a range of established reference scenarios. Our lower bound estimator consistently provides a stable, low-variance estimate of a tight lower bound on MI. We also demonstrate its practical usefulness in the context of the Information Bottleneck framework. Taken together, our results provide new theoretical justifications and strong empirical evidence for using discriminative learning in MI-based representation learning.