# Unified Perspective on Probability Divergence via Maximum Likelihood Density Ratio Estimation: Bridging KL-Divergence and Integral Probability Metrics

arXiv: https://arxiv.org/abs/2201.13127 · HTML: https://mlsanity.com/paper/2201.13127/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 85 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Positive-Unlabeled Learning with Non-Negative Risk Estimator](https://mlsanity.com/paper/1703.00593/) (2017)
- [Telescoping Density-Ratio Estimation](https://mlsanity.com/paper/2006.12204/) (2020)
- [Rethinking Importance Weighting for Deep Learning under Distribution Shift](https://mlsanity.com/paper/2006.04662/) (2020)
- [Generalized Energy Based Models](https://mlsanity.com/paper/2003.05033/) (2020)
- [Wasserstein Dependency Measure for Representation Learning](https://mlsanity.com/paper/1903.11780/) (2019)
- [Dual Discriminator Generative Adversarial Nets](https://mlsanity.com/paper/1709.03831/) (2017)
- [Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimation](https://mlsanity.com/paper/2006.06979/) (2020)
- [From optimal transport to generative modeling: the VEGAN cookbook](https://mlsanity.com/paper/1705.07642/) (2017)
- [KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support](https://mlsanity.com/paper/2106.08929/) (2021)
- [Featurized Density Ratio Estimation](https://mlsanity.com/paper/2107.02212/) (2021)
- [Smoothness and Stability in GANs](https://mlsanity.com/paper/2002.04185/) (2020)
- [Optimal Bounds between f-Divergences and Integral Probability Metrics](https://mlsanity.com/paper/2006.05973/) (2020)
- [Off-Policy Evaluation and Learning for External Validity under a Covariate Shift](https://mlsanity.com/paper/2002.11642/) (2020)
- [Generative Adversarial Nets from a Density Ratio Estimation Perspective](https://mlsanity.com/paper/1610.02920/) (2016)
- [Bridging the Gap Between f-GANs and Wasserstein GANs](https://mlsanity.com/paper/1910.09779/) (2019)
- [Estimating Certain Integral Probability Metric (IPM) is as Hard as Estimating under the IPM](https://mlsanity.com/paper/1911.00730/) (2019)
- [Generative Ratio Matching Networks](https://mlsanity.com/paper/1806.00101/) (2018)
- [Bridging Maximum Likelihood and Adversarial Learning via α-Divergence](https://mlsanity.com/paper/2007.06178/) (2020)
- [Meta-Learning for Relative Density-Ratio Estimation](https://mlsanity.com/paper/2107.00801/) (2021)
- [Relation between the Kantorovich-Wasserstein metric and the Kullback-Leibler divergence](https://mlsanity.com/paper/1908.09211/) (2019)

## Abstract (arXiv)

This paper provides a unified perspective for the Kullback-Leibler (KL)-divergence and the integral probability metrics (IPMs) from the perspective of maximum likelihood density-ratio estimation (DRE). Both the KL-divergence and the IPMs are widely used in various fields in applications such as generative modeling. However, a unified understanding of these concepts has still been unexplored. In this paper, we show that the KL-divergence and the IPMs can be represented as maximal likelihoods differing only by sampling schemes, and use this result to derive a unified form of the IPMs and a relaxed estimation method. To develop the estimation problem, we construct an unconstrained maximum likelihood estimator to perform DRE with a stratified sampling scheme. We further propose a novel class of probability divergences, called the Density Ratio Metrics (DRMs), that interpolates the KL-divergence and the IPMs. In addition to these findings, we also introduce some applications of the DRMs, such as DRE and generative adversarial networks. In experiments, we validate the effectiveness of our proposed methods.