# Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences

arXiv: https://arxiv.org/abs/2402.00501 · HTML: https://mlsanity.com/paper/2402.00501/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 45 references that are in the map.
- First posted on arXiv 1 Feb 2024.

## Builds on (most cited first, 8 of 8)

- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [How much does your data exploration overfit? Controlling bias via information usage](https://mlsanity.com/paper/1511.05219/) (2015)
- [Non-exponentially weighted aggregation: regret bounds for unbounded loss functions](https://mlsanity.com/paper/2009.03017/) (2020)
- [Empirical Risk Minimization with Relative Entropy Regularization](https://mlsanity.com/paper/2211.06617/) (2022)
- [On the Validation of Gibbs Algorithms: Training Datasets, Test Datasets and their Aggregation](https://mlsanity.com/paper/2306.12380/) (2023)
- [Generalization Analysis of Machine Learning Algorithms via the Worst-Case Data-Generating Probability Measure](https://mlsanity.com/paper/2312.12236/) (2023)
- [Analysis of the Relative Entropy Asymmetry in the Regularization of Empirical Risk Minimization](https://mlsanity.com/paper/2306.07123/) (2023)
- [Information-theoretic Analysis of Test Data Sensitivity in Uncertainty](https://mlsanity.com/paper/2307.12456/) (2023)

## Built on by (most cited first, 6 of 6)

- [The Method of Gaps: Exact Expressions for the Generalization Error of Supervised Learning Algorithms](https://mlsanity.com/paper/2411.12030/) (2024)
- [Proofs for Folklore Theorems on the Radon-Nikodym Derivative](https://mlsanity.com/paper/2501.18374/) (2025)
- [Decentralized Machine Learning with Centralized Performance Guarantees via Gibbs Algorithms](https://mlsanity.com/paper/2604.20492/) (2026)
- [A Dual Optimization View to Empirical Risk Minimization with f-Divergence Regularization](https://mlsanity.com/paper/2508.03314/) (2025)
- [Empirical Risk Minimization with f-Divergence Regularization](https://mlsanity.com/paper/2601.13191/) (2026)
- [Machine Unlearning for Gibbs Supervised Learning Algorithms](https://mlsanity.com/paper/2609.29409/) (2026)

## Abstract (arXiv)

The solution to empirical risk minimization with f-divergence regularization (ERM-fDR) is presented under mild conditions on f. Under such conditions, the optimal measure is shown to be unique. Examples of the solution for particular choices of the function f are presented. Previously known solutions to common regularization choices are obtained by leveraging the flexibility of the family of f-divergences. These include the unique solutions to empirical risk minimization with relative entropy regularization (Type-I and Type-II). The analysis of the solution unveils the following properties of f-divergences when used in the ERM-fDR problem: ibigl) f-divergence regularization forces the support of the solution to coincide with the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; and iibigl) any f-divergence regularization is equivalent to a different f-divergence regularization with an appropriate transformation of the empirical risk function.