mlsanityData to 7 Oct 2026

General Loss Functions Lead to (Approximate) Interpolation in High Dimensions

arXiv 2303.07475 ↗First posted 13 Mar 2023stat.ML cs.LG math.OCcited by 4

Not on the map. Why?

Abstract (arXiv)

We provide a unified framework that applies to a general family of convex losses across binary and multiclass settings in the overparameterized regime to approximately characterize the implicit bias of gradient descent in closed form. Specifically, we show that the implicit bias is approximated (but not exactly equal to) the minimum-norm interpolation in high dimensions, which arises from training on the squared loss. In contrast to prior work, which was tailored to exponentially-tailed losses and used the intermediate support-vector-machine formulation, our framework directly builds on the primal-dual analysis of Ji and Telgarsky (2021), allowing us to provide new approximate equivalences for general convex losses through a novel sensitivity analysis. Our framework also recovers existing exact equivalence results for exponentially-tailed losses across binary and multiclass settings. Finally, we provide evidence for the tightness of our techniques and use our results to demonstrate the effect of certain loss functions designed for out-of-distribution problems on the closed-form solution.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 59 references that are in the map. First posted on arXiv 13 Mar 2023.

Builds on

34 of 59 references.

References · 34

  1. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  2. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2020Long-tail learning via logit adjustmentcited by 605
  5. 2019Benign Overfitting in Linear Regressioncited by 544
  6. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  7. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  9. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  10. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  11. 2019Two models of double descent for weak featurescited by 194
  12. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
22 more
  1. 2019Harmless interpolation of noisy data in regressioncited by 121
  2. 2018Convergence of Gradient Descent on Separable Datacited by 108
  3. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  4. 2019A Model of Double Descent for High-dimensional Binary Linear Classificationcited by 81
  5. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  6. 2018A modern maximum-likelihood theory for high-dimensional logistic regressioncited by 79
  7. 2020Identifying and Compensating for Feature Deviation in Imbalanced Deep Learningcited by 76
  8. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  9. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  10. 2017On Loss Functions for Deep Neural Networks in Classificationcited by 68
  11. 2020Gradient descent follows the regularization path for general lossescited by 57
  12. 2018Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularizationcited by 52
  13. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  14. 2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 41
  15. 2019The Impact of Regularization on High-dimensional Logistic Regressioncited by 35
  16. 2020On the proliferation of support vectors in high dimensionscited by 32
  17. 2020Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting and Regularizationcited by 27
  18. 2021Is Importance Weighting Incompatible with Interpolating Classifiers?cited by 19
  19. 2021The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networkscited by 18
  20. 2021Support vector machines and linear regression coincide with very high-dimensional featurescited by 8
  21. 2022Generalization for multiclass classification with overparameterized linear modelscited by 8
  22. 2022On how to avoid exacerbating spurious correlations when models are overparameterizedcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2.

4 papers.

  1. 2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
  2. 2023New Equivalences Between Interpolation and SVMs: Kernels and Structured Featurescited by 3
  3. 2023Precise Asymptotic Generalization for Multiclass Classification with Overparameterized Linear Modelscited by 3
  4. 2024Sharp analysis of out-of-distribution error for "importance-weighted" estimators in the overparameterized regimecited by 3