Data to 14 Oct 2026

Classification vs regression in overparameterized regimes: Does the loss function matter?

arXiv 2005.08054 ↗First posted 16 May 2020cs.LG cs.IT math.ITcited by 97

Not on the map. Why?

Abstract (arXiv)

We compare classification and regression tasks in an overparameterized linear model with Gaussian features. On the one hand, we show that with sufficient overparameterization all training points are support vectors: solutions obtained by least-squares minimum-norm interpolation, typically used for regression, are identical to those produced by the hard-margin support vector machine (SVM) that minimizes the hinge loss, typically used for training classifiers. On the other hand, we show that there exist regimes where these interpolating solutions generalize well when evaluated by the 0-1 test loss function, but do not generalize if evaluated by the square loss function, i.e. they approach the null risk. Our results demonstrate the very different roles and properties of loss functions used at the training phase (optimization) and the testing phase (generalization).

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 34; 2022: 19; 2023: 17; 2024: 15; 2025: 8; 2026: 4.

97 papers.

Sort:

By area:

2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
2023Transformers as Support Vector Machinescited by 74
2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 57
2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 49
2021Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolationcited by 42
2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
2022The Implicit Bias of Benign Overfittingcited by 38
2021Selective Regression Under Fairness Criteriacited by 34
2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34

38 more

2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 30
2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 26
2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
2022The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspectivecited by 24
2021Interpolating Classifiers Make Few Mistakescited by 22
2022Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Biascited by 22
2022Towards Sample-efficient Overparameterized Meta-learningcited by 21
2021RATT: Leveraging Unlabeled Data to Guarantee Generalizationcited by 20
2021Towards Understanding the Data Dependency of Mixup-style Trainingcited by 19
2021Is Importance Weighting Incompatible with Interpolating Classifiers?cited by 19
2022SphereFed: Hyperspherical Federated Learningcited by 19
2021The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networkscited by 18
2022A Blessing of Dimensionality in Membership Inference through Regularizationcited by 18
2021Understanding and Mitigating Accuracy Disparity in Regressioncited by 17
2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17
2021Tight bounds for minimum l1-norm interpolation of noisy datacited by 16
2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
2024Provable Weak-to-Strong Generalization via Benign Overfittingcited by 16
2021Interpolation can hurt robust generalization even when there is no noisecited by 15
2024Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smootherscited by 15
2021Harmless interpolation in regression and classification with structured featurescited by 14
2021Understanding Square Loss in Training Overparametrized Neural Network Classifierscited by 14
2023A Unified Approach to Controlling Implicit Regularization via Mirror Descentcited by 14
2024Implicit Optimization Bias of Next-Token Prediction in Linear Modelscited by 14
2021On the interplay between data structure and loss function in classification problemscited by 13
2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 12
2021Information-Theoretic Characterization of the Generalization Error for Iterative Semi-Supervised Learningcited by 11
2022A Non-Asymptotic Moreau Envelope Theory for High-Dimensional Generalized Linear Modelscited by 11
2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
2021Reconstruction on Trees and Low-Degree Polynomialscited by 10
2025Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictionscited by 10
2024The Surprising Harmfulness of Benign Overfitting for Adversarial Robustnesscited by 9
2025Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regressioncited by 9
2021Support vector machines and linear regression coincide with very high-dimensional featurescited by 8
2022Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensionscited by 8
2022Generalization for multiclass classification with overparameterized linear modelscited by 8

50 most cited of 97 All 97 (JSON)