mlsanityData to 7 Oct 2026

Practical and Consistent Estimation of f-Divergences

arXiv 1905.11112 ↗First posted 27 May 2019stat.ML cs.IT cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

The estimation of an f-divergence between two probability distributions based on samples is a fundamental problem in statistics and machine learning. Most works study this problem under very weak assumptions, in which case it is provably hard. We consider the case of stronger structural assumptions that are commonly satisfied in modern machine learning, including representation learning and generative modelling with autoencoder architectures. Under these assumptions we propose and study an estimator that can be easily implemented, works well in high dimensions, and enjoys faster rates of convergence. We verify the behavior of our estimator empirically in both synthetic and real-data experiments, and discuss its direct implications for total correlation, entropy, and mutual information estimation.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 8; 2023: 3; 2024: 4; 2025: 3; 2026: 3.

24 papers.

Sort:

By area:

  1. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  2. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
  3. 2022PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Taskscited by 30
  4. 2021On the Properties of Kullback-Leibler Divergence Between Multivariate Gaussian Distributionscited by 22
  5. 2021Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integralscited by 18
  6. 2021Martingale Methods for Sequential Estimation of Convex Functionals and Divergencescited by 17
  7. 2022Learning Disentangled Textual Representations via Statistical Measures of Similaritycited by 14
  8. 2023Performative Prediction with Bandit Feedback: Learning through Reparameterizationcited by 11
  9. 2024Discriminative Estimation of Total Variation Distance: A Fidelity Auditor for Generative Datacited by 10
  10. 2022RenyiCL: Contrastive Representation Learning with Skew Renyi Divergencecited by 8
  11. 2022Striking a Balance: Alleviating Inconsistency in Pre-trained Models for Symmetric Classification Taskscited by 6
  12. 2022Sum-of-Squares Relaxations for Information Theory and Variational Inferencecited by 6
12 more
  1. 2025Explicit Preference Optimization: No Need for an Implicit Reward Modelcited by 6
  2. 2024New Desiderata for Direct Preference Optimizationcited by 5
  3. 2025RSPO: Regularized Self-Play Alignment of Large Language Modelscited by 5
  4. 2024Robust Generative Learning with Lipschitz-Regularized α-Divergences Allows Minimal Assumptions on Target Distributionscited by 4
  5. 2026When Environments Shift: Safe Planning with Generative Priors and Robust Conformal Predictioncited by 2
  6. 2026Weight Clipping for Robust Conformal Inference under Unbounded Covariate Shiftscited by 2
  7. 2022Collaborative likelihood-ratio estimation over graphscited by 1
  8. 2023Online non-parametric likelihood-ratio estimation by Pearson-divergence functional minimizationcited by 1
  9. 2022KL Divergence Estimation with Multi-group Attributioncited by 0
  10. 2023Stabilizing Subject Transfer in EEG Classification with Divergence Estimationcited by 0
  11. 2025On approximating the f-divergence between two Ising modelscited by 0
  12. 2026Approximating f-Divergences with Rank Statisticscited by 0