# Guaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalities

arXiv: https://arxiv.org/abs/1606.05850 · HTML: https://mlsanity.com/paper/1606.05850/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2016.

## Built on by (most cited first, 16 of 16)

- [Out-of-Candidate Rectification for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2211.12268/) (2022)
- [TEA: Test-time Energy Adaptation](https://mlsanity.com/paper/2311.14402/) (2023)
- [Fuzzy Positive Learning for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2210.08519/) (2022)
- [Convex Bounds on the Softmax Function with Applications to Robustness Verification](https://mlsanity.com/paper/2303.01713/) (2023)
- [MuMIC -- Multimodal Embedding for Multi-label Image Classification with Tempered Sigmoid](https://mlsanity.com/paper/2211.05232/) (2022)
- [Improved Stochastic Optimization of LogSumExp](https://mlsanity.com/paper/2509.24894/) (2025)
- [Fast approximations of the Jeffreys divergence between univariate Gaussian mixture models via exponential polynomial densities](https://mlsanity.com/paper/2107.05901/) (2021)
- [Multi-Fidelity Bayesian Optimization With Across-Task Transferable Max-Value Entropy Search](https://mlsanity.com/paper/2403.09570/) (2024)
- [Theoretical Error Analysis of Entropy Approximation for Gaussian Mixtures](https://mlsanity.com/paper/2202.13059/) (2022)
- [T5Score: A Methodology for Automatically Assessing the Quality of LLM Generated Multi-Document Topic Sets](https://mlsanity.com/paper/2407.17390/) (2024)
- [A Transductive Maximum Margin Classifier for Few-Shot Learning](https://mlsanity.com/paper/2107.11975/) (2021)
- [Spatial Envelope MPC: High Performance Driving without a Reference](https://mlsanity.com/paper/2509.18506/) (2025)
- [Intra-Class Subdivision for Pixel Contrastive Learning: Application to Semi-supervised Cardiac Image Segmentation](https://mlsanity.com/paper/2602.00174/) (2026)
- [Deep Arguing](https://mlsanity.com/paper/2605.10569/) (2026)
- [Quantitative Linear Logic for Neuro-Symbolic Learning and Verification](https://mlsanity.com/paper/2605.13845/) (2026)
- [RIMS: Preference Optimization via Smoothed Multi-pair Aggregation for Small-Scale LLM Retrieval-Augmented Generation](https://mlsanity.com/paper/2607.16431/) (2026)

## Abstract (arXiv)

Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using costly Monte-Carlo stochastic integration, approximated, or bounded using various techniques. We present a fast and generic method that builds algorithmically closed-form lower and upper bounds on the entropy, the cross-entropy and the Kullback-Leibler divergence of mixtures. We illustrate the versatile method by reporting on our experiments for approximating the Kullback-Leibler divergence between univariate exponential mixtures, Gaussian mixtures, Rayleigh mixtures, and Gamma mixtures.