mlsanityData to 7 Oct 2026

Regression as Classification: Influence of Task Formulation on Neural Network Features

arXiv 2211.05641 ↗First posted 10 Nov 2022cs.LG cs.AI stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

Neural networks can be trained to solve regression problems by using gradient-based methods to minimize the square loss. However, practitioners often prefer to reformulate regression as a classification problem, observing that training on the cross entropy loss results in better performance. By focusing on two-layer ReLU networks, which can be fully characterized by measures over their feature space, we explore how the implicit bias induced by gradient-based optimization could partly explain the above phenomenon. We provide theoretical evidence that the regression formulation yields a measure whose support can differ greatly from that for classification, in the case of one-dimensional data. Our proposed optimal supports correspond directly to the features learned by the input layer of the network. The different nature of these supports sheds light on possible optimization difficulties the square loss could encounter during training, and we present empirical results illustrating this phenomenon.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 29 of its 48 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

29 of 48 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2021Convex optimizationcited by 2,479
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  9. 2016Colorful Image Colorizationcited by 1,113
  10. 2016Pixel Recurrent Neural Networkscited by 814
  11. 2019Solving Rubik's Cube with a Robot Handcited by 699
  12. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
17 more
  1. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  2. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  3. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  4. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  5. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  6. 2013DeepPose: Human Pose Estimation via Deep Neural Networkscited by 305
  7. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  8. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  9. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  10. 2014Qualitatively characterizing neural network optimization problemscited by 186
  11. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  12. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  13. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  14. 2013A Deep and Tractable Density Estimatorcited by 67
  15. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  16. 2015Generative Image Modeling Using Spatial LSTMscited by 35
  17. 2015Robust Optimization for Deep Regressioncited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 13; 2025: 10; 2026: 2.

29 papers.

Sort:

By area:

  1. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  2. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  3. 2024FoldToken: Learning Protein Language via Vector Quantization and Beyondcited by 21
  4. 2024Conformal Prediction via Regression-as-Classificationcited by 18
  5. 2023Penalising the biases in norm regularisation enforces sparsitycited by 17
  6. 2023Pathologies of Predictive Diversity in Deep Ensemblescited by 16
  7. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  8. 2024GeoPlant: Spatial Plant Species Prediction Datasetcited by 16
  9. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  10. 2025An overview of condensation phenomenon in deep learningcited by 13
  11. 2024Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domainscited by 12
  12. 2024Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Modelscited by 12
17 more
  1. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  2. 2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
  3. 2024An Efficient Model-Agnostic Approach for Uncertainty Estimation in Data-Restricted Pedometric Applicationscited by 3
  4. 2025Reliable uncertainty quantification for 2D/3D anatomical landmark localization using multi-output conformal predictioncited by 2
  5. 2025The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representationscited by 2
  6. 2025Learning Task-Agnostic Representations through Multi-Teacher Distillationcited by 2
  7. 2024Average-Over-Time Spiking Neural Networks for Uncertainty Estimation in Regressioncited by 1
  8. 2024Structural Entropy Guided Probabilistic Codingcited by 1
  9. 2025Building Bridges between Regression, Clustering, and Classificationcited by 1
  10. 2025CARIL: Confidence-Aware Regression in Imitation Learning for Autonomous Drivingcited by 1
  11. 2026When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Modelscited by 1
  12. 2023Target Variable Engineeringcited by 0
  13. 2024The Effects of Multi-Task Learning on ReLU Neural Network Functionscited by 0
  14. 2025BoundarySeg:Boundary-Aware Multi-Task Learning for Semi-Supervised 3D Medical Image Segmentation with Limited Unlabeled Datacited by 0
  15. 2025Why Regression? Binary Encoding Classification Brings Confidence to Stock Market Index Price Predictioncited by 0
  16. 2025Methodological Insights into Structural Causal Modelling and Uncertainty-Aware Forecasting for Economic Indicatorscited by 0
  17. 2026Altitude-Adaptive Vision-Only Geo-Localization for UAVs in GPS-Denied Environmentscited by 0