# Fast approximations of the Jeffreys divergence between univariate Gaussian mixture models via exponential polynomial densities

arXiv: https://arxiv.org/abs/2107.05901 · HTML: https://mlsanity.com/paper/2107.05901/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 138 references that are in the map.
- First posted on arXiv 13 Jul 2021.

## Builds on (most cited first, 16 of 16)

- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Offline Reinforcement Learning with Fisher Divergence Critic Regularization](https://mlsanity.com/paper/2103.08050/) (2021)
- [An elementary introduction to information geometry](https://mlsanity.com/paper/1808.08271/) (2018)
- [On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means](https://mlsanity.com/paper/1904.04017/) (2019)
- [Relative Loss Bounds for On-line Density Estimation with the Exponential Family of Distributions](https://mlsanity.com/paper/1301.6677/) (2013)
- [Guaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalities](https://mlsanity.com/paper/1606.05850/) (2016)
- [Non-Asymptotic Performance Guarantees for Neural Estimation of mathsff-Divergences](https://mlsanity.com/paper/2103.06923/) (2021)
- [Variational approximations using Fisher divergence](https://mlsanity.com/paper/1905.05284/) (2019)
- [Change Detection in Multivariate Datastreams: Likelihood and Detectability Loss](https://mlsanity.com/paper/1510.04850/) (2015)
- [Maximum Number of Modes of Gaussian Mixtures](https://mlsanity.com/paper/1702.05066/) (2017)
- [Guaranteed Deterministic Bounds on the Total Variation Distance between Univariate Mixtures](https://mlsanity.com/paper/1806.11311/) (2018)
- [On H"older projective divergences](https://mlsanity.com/paper/1701.03916/) (2017)
- [Fisher Auto-Encoders](https://mlsanity.com/paper/2007.06120/) (2020)
- [Maximum Likelihood Estimates for Gaussian Mixtures Are Transcendental](https://mlsanity.com/paper/1508.06958/) (2015)
- [A series of maximum entropy upper bounds of the differential entropy](https://mlsanity.com/paper/1612.02954/) (2016)
- [The statistical Minkowski distances: Closed-form formula for Gaussian Mixture Models](https://mlsanity.com/paper/1901.03732/) (2019)

## Built on by (most cited first, 3 of 3)

- [Statistical Efficiency of Score Matching: The View from Isoperimetry](https://mlsanity.com/paper/2210.00726/) (2022)
- [IIFL: Implicit Interactive Fleet Learning from Heterogeneous Human Supervisors](https://mlsanity.com/paper/2306.15228/) (2023)
- [Divergences induced by dual subtractive and divisive normalizations of exponential families and their convex deformations](https://mlsanity.com/paper/2312.12849/) (2023)

## Abstract (arXiv)

The Jeffreys divergence is a renown symmetrization of the oriented Kullback-Leibler divergence broadly used in information sciences. Since the Jeffreys divergence between Gaussian mixture models is not available in closed-form, various techniques with pros and cons have been proposed in the literature to either estimate, approximate, or lower and upper bound this divergence. In this paper, we propose a simple yet fast heuristic to approximate the Jeffreys divergence between two univariate Gaussian mixtures with arbitrary number of components. Our heuristic relies on converting the mixtures into pairs of dually parameterized probability densities belonging to an exponential family. In particular, we consider the versatile polynomial exponential family densities, and design a divergence to measure in closed-form the goodness of fit between a Gaussian mixture and its polynomial exponential density approximation. This goodness-of-fit divergence is a generalization of the Hyv"arinen divergence used to estimate models with computationally intractable normalizers. It allows us to perform model selection by choosing the orders of the polynomial exponential densities used to approximate the mixtures. We demonstrate experimentally that our heuristic to approximate the Jeffreys divergence improves by several orders of magnitude the computational time of stochastic Monte Carlo estimations while approximating reasonably well the Jeffreys divergence, specially when the mixtures have a very small number of modes. Besides, our mixture-to-exponential family conversion techniques may prove useful in other settings.