mlsanityData to 7 Oct 2026

Functional Risk Minimization

arXiv 2412.21149 ↗First posted 30 Dec 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The field of Machine Learning has changed significantly since the 1970s. However, its most basic principle, Empirical Risk Minimization (ERM), remains unchanged. We propose Functional Risk Minimization~(FRM), a general framework where losses compare functions rather than outputs. This results in better performance in supervised, unsupervised, and RL experiments. In the FRM paradigm, for each data point (xi,yi) there is function fθi that fits it: yi = fθi(xi). This allows FRM to subsume ERM for many common loss functions and to capture more realistic noise processes. We also show that FRM provides an avenue towards understanding generalization in the modern over-parameterized regime, as its objective can be framed as finding the simplest model that fits the training data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 30 Dec 2024.

Builds on

23 of 34 references.

References · 23

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2019Invariant Risk Minimizationcited by 1,832
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  8. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  9. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  10. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  11. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  12. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
11 more
  1. 2019Meta-Learning with Implicit Gradientscited by 523
  2. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  3. 2017Noisy Networks for Explorationcited by 262
  4. 2021Deep learning: a statistical viewpointcited by 239
  5. 2021Efficient and Modular Implicit Differentiationcited by 199
  6. 2018Recasting Gradient-Based Meta-Learning as Hierarchical Bayescited by 180
  7. 2019Test-Time Training with Self-Supervision for Generalization under Distribution Shiftscited by 55
  8. 2017Theory of Deep Learning III: explaining the non-overfitting puzzlecited by 32
  9. 2021Noether Networks: Meta-Learning Useful Conserved Quantitiescited by 26
  10. 2019Integrals over Gaussians under Linear Domain Constraintscited by 15
  11. 2020Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction timecited by 13

Built on by

Not cited yet.