mlsanityData to 7 Oct 2026

Unified Perspective on Probability Divergence via Maximum Likelihood Density Ratio Estimation: Bridging KL-Divergence and Integral Probability Metrics

arXiv 2201.13127 ↗First posted 31 Jan 2022cs.LG cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper provides a unified perspective for the Kullback-Leibler (KL)-divergence and the integral probability metrics (IPMs) from the perspective of maximum likelihood density-ratio estimation (DRE). Both the KL-divergence and the IPMs are widely used in various fields in applications such as generative modeling. However, a unified understanding of these concepts has still been unexplored. In this paper, we show that the KL-divergence and the IPMs can be represented as maximal likelihoods differing only by sampling schemes, and use this result to derive a unified form of the IPMs and a relaxed estimation method. To develop the estimation problem, we construct an unconstrained maximum likelihood estimator to perform DRE with a stratified sampling scheme. We further propose a novel class of probability divergences, called the Density Ratio Metrics (DRMs), that interpolates the KL-divergence and the IPMs. In addition to these findings, we also introduce some applications of the DRMs, such as DRE and generative adversarial networks. In experiments, we validate the effectiveness of our proposed methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 85 references that are in the map. First posted on arXiv 31 Jan 2022.

Builds on

24 of 85 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  3. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  4. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  5. 2017Positive-Unlabeled Learning with Non-Negative Risk Estimatorcited by 217
  6. 2020Telescoping Density-Ratio Estimationcited by 108
  7. 2020Rethinking Importance Weighting for Deep Learning under Distribution Shiftcited by 92
  8. 2020Generalized Energy Based Modelscited by 73
  9. 2019Wasserstein Dependency Measure for Representation Learningcited by 69
  10. 2017Dual Discriminator Generative Adversarial Netscited by 55
  11. 2020Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimationcited by 43
  12. 2017From optimal transport to generative modeling: the VEGAN cookbookcited by 41
12 more
  1. 2021KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Supportcited by 37
  2. 2021Featurized Density Ratio Estimationcited by 33
  3. 2020Smoothness and Stability in GANscited by 27
  4. 2020Optimal Bounds between f-Divergences and Integral Probability Metricscited by 27
  5. 2020Off-Policy Evaluation and Learning for External Validity under a Covariate Shiftcited by 26
  6. 2016Generative Adversarial Nets from a Density Ratio Estimation Perspectivecited by 24
  7. 2019Bridging the Gap Between f-GANs and Wasserstein GANscited by 23
  8. 2019Estimating Certain Integral Probability Metric (IPM) is as Hard as Estimating under the IPMcited by 11
  9. 2018Generative Ratio Matching Networkscited by 6
  10. 2020Bridging Maximum Likelihood and Adversarial Learning via α-Divergencecited by 6
  11. 2021Meta-Learning for Relative Density-Ratio Estimationcited by 6
  12. 2019Relation between the Kantorovich-Wasserstein metric and the Kullback-Leibler divergencecited by 4

Built on by

Not cited yet.