# Regression as Classification: Influence of Task Formulation on Neural Network Features

arXiv: https://arxiv.org/abs/2211.05641 · HTML: https://mlsanity.com/paper/2211.05641/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 10 Nov 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1412.6614/) (2014)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Gradient Descent Maximizes the Margin of Homogeneous Neural Networks](https://mlsanity.com/paper/1906.05890/) (2019)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020)
- [Characterizing Implicit Bias in Terms of Optimization Geometry](https://mlsanity.com/paper/1802.08246/) (2018)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [Gradient descent aligns the layers of deep linear networks](https://mlsanity.com/paper/1810.02032/) (2018)
- [Spurious Local Minima are Common in Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/1712.08968/) (2017)
- [Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs](https://mlsanity.com/paper/2206.00939/) (2022)
- [A Deep and Tractable Density Estimator](https://mlsanity.com/paper/1310.1757/) (2013)
- [Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models](https://mlsanity.com/paper/1905.07325/) (2019)
- [Generative Image Modeling Using Spatial LSTMs](https://mlsanity.com/paper/1506.03478/) (2015)
- [Robust Optimization for Deep Regression](https://mlsanity.com/paper/1505.06606/) (2015)

## Built on by (most cited first, 29 of 29)

- [Stop Regressing: Training Value Functions via Classification for Scalable Deep RL](https://mlsanity.com/paper/2403.03950/) (2024)
- [Disentangling the Causes of Plasticity Loss in Neural Networks](https://mlsanity.com/paper/2402.18762/) (2024)
- [FoldToken: Learning Protein Language via Vector Quantization and Beyond](https://mlsanity.com/paper/2403.09673/) (2024)
- [Penalising the biases in norm regularisation enforces sparsity](https://mlsanity.com/paper/2303.01353/) (2023)
- [Conformal Prediction via Regression-as-Classification](https://mlsanity.com/paper/2404.08168/) (2024)
- [An overview of condensation phenomenon in deep learning](https://mlsanity.com/paper/2504.09484/) (2025)
- [Pathologies of Predictive Diversity in Deep Ensembles](https://mlsanity.com/paper/2302.00704/) (2023)
- [Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs](https://mlsanity.com/paper/2306.06479/) (2023)
- [GeoPlant: Spatial Plant Species Prediction Dataset](https://mlsanity.com/paper/2408.13928/) (2024)
- [Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models](https://mlsanity.com/paper/2411.02083/) (2024)
- [Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild](https://mlsanity.com/paper/2503.10065/) (2025)
- [Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domains](https://mlsanity.com/paper/2405.07414/) (2024)
- [Simplicity bias and optimization threshold in two-layer ReLU networks](https://mlsanity.com/paper/2410.02348/) (2024)
- [Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?](https://mlsanity.com/paper/2406.06309/) (2024)
- [Learning Task-Agnostic Representations through Multi-Teacher Distillation](https://mlsanity.com/paper/2510.18680/) (2025)
- [An Efficient Model-Agnostic Approach for Uncertainty Estimation in Data-Restricted Pedometric Applications](https://mlsanity.com/paper/2409.11985/) (2024)
- [Reliable uncertainty quantification for 2D/3D anatomical landmark localization using multi-output conformal prediction](https://mlsanity.com/paper/2503.14106/) (2025)
- [Building Bridges between Regression, Clustering, and Classification](https://mlsanity.com/paper/2502.02996/) (2025)
- [The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representations](https://mlsanity.com/paper/2507.05644/) (2025)
- [Average-Over-Time Spiking Neural Networks for Uncertainty Estimation in Regression](https://mlsanity.com/paper/2412.00278/) (2024)
- [Structural Entropy Guided Probabilistic Coding](https://mlsanity.com/paper/2412.08841/) (2024)
- [CARIL: Confidence-Aware Regression in Imitation Learning for Autonomous Driving](https://mlsanity.com/paper/2503.00783/) (2025)
- [When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Models](https://mlsanity.com/paper/2601.11444/) (2026)
- [Target Variable Engineering](https://mlsanity.com/paper/2310.09440/) (2023)
- [The Effects of Multi-Task Learning on ReLU Neural Network Functions](https://mlsanity.com/paper/2410.21696/) (2024)
- [BoundarySeg:Boundary-Aware Multi-Task Learning for Semi-Supervised 3D Medical Image Segmentation with Limited Unlabeled Data](https://mlsanity.com/paper/2505.09829/) (2025)
- [Why Regression? Binary Encoding Classification Brings Confidence to Stock Market Index Price Prediction](https://mlsanity.com/paper/2506.03153/) (2025)
- [Methodological Insights into Structural Causal Modelling and Uncertainty-Aware Forecasting for Economic Indicators](https://mlsanity.com/paper/2509.07036/) (2025)
- [Altitude-Adaptive Vision-Only Geo-Localization for UAVs in GPS-Denied Environments](https://mlsanity.com/paper/2602.23872/) (2026)

## Abstract (arXiv)

Neural networks can be trained to solve regression problems by using gradient-based methods to minimize the square loss. However, practitioners often prefer to reformulate regression as a classification problem, observing that training on the cross entropy loss results in better performance. By focusing on two-layer ReLU networks, which can be fully characterized by measures over their feature space, we explore how the implicit bias induced by gradient-based optimization could partly explain the above phenomenon. We provide theoretical evidence that the regression formulation yields a measure whose support can differ greatly from that for classification, in the case of one-dimensional data. Our proposed optimal supports correspond directly to the features learned by the input layer of the network. The different nature of these supports sheds light on possible optimization difficulties the square loss could encounter during training, and we present empirical results illustrating this phenomenon.