mlsanityData to 14 Oct 2026

Asymmetry of the Relative Entropy in the Regularization of Empirical Risk Minimization

arXiv 2410.02833 ↗First posted 2 Oct 2024stat.ML cs.IT cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

The effect of relative entropy asymmetry is analyzed in the context of empirical risk minimization (ERM) with relative entropy regularization (ERM-RER). Two regularizations are considered: (a) the relative entropy of the measure to be optimized with respect to a reference measure (Type-I ERM-RER); and (b) the relative entropy of the reference measure with respect to the measure to be optimized (Type-II ERM-RER). The main result is the characterization of the solution to the Type-II ERM-RER problem and its key properties. By comparing the well-understood Type-I ERM-RER with Type-II ERM-RER, the effects of entropy asymmetry are highlighted. The analysis shows that in both cases, regularization by relative entropy forces the solution's support to collapse into the support of the reference measure, introducing a strong inductive bias that negates the evidence provided by the training data. Finally, it is shown that Type-II regularization is equivalent to Type-I regularization with an appropriate transformation of the empirical risk function.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 18 of its 81 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

18 of 81 references.

References · 18

  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  2. 2017A Closer Look at Memorization in Deep Networkscited by 942
  3. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  4. 2023Generalization Bounds: Perspectives from Information Theory and PAC-Bayescited by 61
  5. 2022Rashomon Capacity: A Metric for Predictive Multiplicity in Classificationcited by 47
  6. 2020Non-exponentially weighted aggregation: regret bounds for unbounded loss functionscited by 18
  7. 2022Empirical Risk Minimization with Relative Entropy Regularizationcited by 13
  8. 2019A Tunable Loss Function for Binary Classificationcited by 11
  9. 2022On the Generalization for Transfer Learning: An Information-Theoretic Analysiscited by 11
  10. 2021Realizing GANs via a Tunable Loss Functioncited by 10
  11. 2023On the Validation of Gibbs Algorithms: Training Datasets, Test Datasets and their Aggregationcited by 10
  12. 2024An Information-Theoretic Approach to Generalization Theorycited by 9
6 more
  1. 2022Empirical Risk Minimization with Relative Entropy Regularization: Optimality and Sensitivity Analysiscited by 8
  2. 2023Generalization Analysis of Machine Learning Algorithms via the Worst-Case Data-Generating Probability Measurecited by 7
  3. 2022From Generalisation Error to Transportation-cost Inequalities and Backcited by 6
  4. 2016The Recycling Gibbs Sampler for Efficient Learningcited by 5
  5. 2023Analysis of the Relative Entropy Asymmetry in the Regularization of Empirical Risk Minimizationcited by 5
  6. 2014An Accelerated Proximal Coordinate Gradient Method and its Application to Regularized Empirical Risk Minimizationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 3.

6 papers.

  1. 2024The Method of Gaps: Exact Expressions for the Generalization Error of Supervised Learning Algorithmscited by 5
  2. 2025Proofs for Folklore Theorems on the Radon-Nikodym Derivativecited by 4
  3. 2026Decentralized Machine Learning with Centralized Performance Guarantees via Gibbs Algorithmscited by 2
  4. 2025A Dual Optimization View to Empirical Risk Minimization with f-Divergence Regularizationcited by 1
  5. 2026Empirical Risk Minimization with f-Divergence Regularizationcited by 0
  6. 2026Machine Unlearning for Gibbs Supervised Learning Algorithmscited by 0