mlsanityData to 7 Oct 2026

Empirical Risk Minimization with f-Divergence Regularization

arXiv 2601.13191 ↗First posted 19 Jan 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, the solution to the empirical risk minimization problem with f-divergence regularization (ERM-fDR) is presented and conditions under which the solution also serves as the solution to the minimization of the expected empirical risk subject to an f-divergence constraint are established. The proposed approach extends applicability to a broader class of f-divergences than previously reported and yields theoretical results that recover previously known results. Additionally, the difference between the expected empirical risk of the ERM-fDR solution and that of its reference measure is characterized, providing insights into previously studied cases of f-divergences. A central contribution is the introduction of the normalization function, a mathematical object that is critical in both the dual formulation and practical computation of the ERM-fDR solution. This work presents an implicit characterization of the normalization function as a nonlinear ordinary differential equation (ODE), establishes its key properties, and subsequently leverages them to construct a numerical algorithm for approximating the normalization factor under mild assumptions. Further analysis demonstrates structural equivalences between ERM-fDR problems with different f-divergences via transformations of the empirical risk. Finally, the proposed algorithm is used to compute the training and test risks of ERM-fDR solutions under different f-divergence regularizers. This numerical example highlights the practical implications of choosing different functions f in ERM-fDR problems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 85 references that are in the map. First posted on arXiv 19 Jan 2026.

Builds on

36 of 85 references.

References · 36

  1. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  2. 2017User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradientcited by 116
  3. 2017Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimizationcited by 96
  4. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  5. 2017Convergence of Langevin MCMC in KL-divergencecited by 86
  6. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  7. 2015Sliced Wasserstein Kernels for Probability Distributionscited by 64
  8. 2019Generalization Error Bounds Via R'enyi-, f-Divergences and Maximal Leakagecited by 48
  9. 2020When Optimizing f-divergence is Robust with Label Noisecited by 44
  10. 2020PAC-Bayes unleashed: generalisation bounds with unbounded lossescited by 43
  11. 2020Jensen-Shannon Information Based Characterization of the Generalization Error of Learning Algorithmscited by 19
  12. 2020Non-exponentially weighted aggregation: regret bounds for unbounded loss functionscited by 18
24 more
  1. 2019Distribution-Dependent Analysis of Gibbs-ERM Principlecited by 17
  2. 2023Mutual Information Estimation via f-Divergence and Data Derangementscited by 17
  3. 2019Variational f-divergence Minimizationcited by 16
  4. 2019Still no free lunches: the price to pay for tighter PAC-Bayes boundscited by 14
  5. 2022Empirical Risk Minimization with Relative Entropy Regularizationcited by 13
  6. 2022Tighter Expected Generalization Error Bounds via Convexity of Information Measurescited by 11
  7. 2022On the Generalization for Transfer Learning: An Information-Theoretic Analysiscited by 11
  8. 2023On the Validation of Gibbs Algorithms: Training Datasets, Test Datasets and their Aggregationcited by 10
  9. 2024f-Divergence Based Classification: Beyond the Use of Cross-Entropycited by 8
  10. 2021Stochastic Gradient Langevin Dynamics with Variance Reductioncited by 7
  11. 2022f-divergences and their applications in lossy compression and bounding generalization errorcited by 7
  12. 2023Generalization Analysis of Machine Learning Algorithms via the Worst-Case Data-Generating Probability Measurecited by 7
  13. 2024Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergencescited by 6
  14. 2024Asymmetry of the Relative Entropy in the Regularization of Empirical Risk Minimizationcited by 6
  15. 2016The Recycling Gibbs Sampler for Efficient Learningcited by 5
  16. 2023Analysis of the Relative Entropy Asymmetry in the Regularization of Empirical Risk Minimizationcited by 5
  17. 2025Proofs for Folklore Theorems on the Radon-Nikodym Derivativecited by 4
  18. 2025How good is PAC-Bayes at explaining generalisation?cited by 4
  19. 2025On Weak-to-Strong Generalization and f-Divergencecited by 4
  20. 2023Information-theoretic Analysis of Test Data Sensitivity in Uncertaintycited by 3
  21. 2025Regularization via f-Divergence: An Application to Multi-Oxide Spectroscopic Analysiscited by 3
  22. 2025Robust Classification with Noisy Labels Based on Posterior Maximizationcited by 2
  23. 2025An Information-Theoretic Analysis for Federated Learning under Concept Driftcited by 2
  24. 2025A Dual Optimization View to Empirical Risk Minimization with f-Divergence Regularizationcited by 1

Built on by

Not cited yet.