# High-dimensional robust regression under heavy-tailed data: Asymptotics and Universality

arXiv: https://arxiv.org/abs/2309.16476 · HTML: https://mlsanity.com/paper/2309.16476/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 60 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 22 of 22)

- [Mean estimation and regression under heavy-tailed distributions--a survey](https://mlsanity.com/paper/1906.04280/) (2019)
- [Learning curves of generic features maps for realistic datasets with a teacher-student model](https://mlsanity.com/paper/2102.08127/) (2021)
- [Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models](https://mlsanity.com/paper/1708.03395/) (2017)
- [A modern maximum-likelihood theory for high-dimensional logistic regression](https://mlsanity.com/paper/1803.06964/) (2018)
- [Adaptive Huber Regression](https://mlsanity.com/paper/1706.06991/) (2017)
- [Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions](https://mlsanity.com/paper/2106.03791/) (2021)
- [Loss minimization and parameter estimation with heavy tails](https://mlsanity.com/paper/1307.1827/) (2013)
- [Robust regression with covariate filtering: Heavy tails and adversarial contamination](https://mlsanity.com/paper/2009.12976/) (2020)
- [The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression](https://mlsanity.com/paper/1804.09753/) (2018)
- [Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification](https://mlsanity.com/paper/2102.07856/) (2021)
- [De-biasing the Lasso: Optimal Sample Size for Gaussian Designs](https://mlsanity.com/paper/1508.02757/) (2015)
- [Gaussian Universality of Perceptrons with Random Labels](https://mlsanity.com/paper/2205.13303/) (2022)
- [Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimation](https://mlsanity.com/paper/2302.08923/) (2023)
- [Theoretical characterization of uncertainty in high-dimensional linear classification](https://mlsanity.com/paper/2202.03295/) (2022)
- [On Empirical Risk Minimization with Dependent and Heavy-Tailed Data](https://mlsanity.com/paper/2109.02224/) (2021)
- [On double-descent in uncertainty quantification in overparametrized models](https://mlsanity.com/paper/2210.12760/) (2022)
- [The Asymptotic Distribution of the MLE in High-dimensional Logistic Models: Arbitrary Covariance](https://mlsanity.com/paper/2001.09351/) (2020)
- [Out-of-sample error estimate for robust M-estimators with convex penalty](https://mlsanity.com/paper/2008.11840/) (2020)
- [Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approach](https://mlsanity.com/paper/2304.02912/) (2023)
- [Robust and Sparse Estimation of Linear Regression Coefficients with Heavy-tailed Noises and Covariates](https://mlsanity.com/paper/2206.07594/) (2022)
- [Asymptotic normality of robust M-estimators with convex penalty](https://mlsanity.com/paper/2107.03826/) (2021)
- [Asymptotic Characterisation of Robust Empirical Risk Minimisation Performance in the Presence of Outliers](https://mlsanity.com/paper/2305.18974/) (2023)

## Built on by (most cited first, 15 of 15)

- [Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training](https://mlsanity.com/paper/2505.17638/) (2025)
- [Asymptotics of feature learning in two-layer networks after one gradient-step](https://mlsanity.com/paper/2402.04980/) (2024)
- [A solvable model of learning generative diffusion: theory and insights](https://mlsanity.com/paper/2501.03937/) (2025)
- [Asymptotics of Non-Convex Generalized Linear Models in High-Dimensions: A proof of the replica formula](https://mlsanity.com/paper/2502.20003/) (2025)
- [High-dimensional learning of narrow neural networks](https://mlsanity.com/paper/2409.13904/) (2024)
- [A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offs](https://mlsanity.com/paper/2402.05674/) (2024)
- [A non-asymptotic distributional theory of approximate message passing for sparse and robust regression](https://mlsanity.com/paper/2401.03923/) (2024)
- [On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds](https://mlsanity.com/paper/2410.16073/) (2024)
- [On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classification](https://mlsanity.com/paper/2506.12454/) (2025)
- [Breaking the curse of dimensionality for linear rules: optimal predictors over the ellipsoid](https://mlsanity.com/paper/2509.21174/) (2025)
- [Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimization](https://mlsanity.com/paper/2604.03146/) (2026)
- [Transfer Learning in ℓ1 Regularized Regression: Hyperparameter Selection Strategy based on Sharp Asymptotic Analysis](https://mlsanity.com/paper/2409.17704/) (2024)
- [Heavy-Tailed Principal Component Analysis](https://mlsanity.com/paper/2603.11308/) (2026)
- [The Conjugate Domain Dichotomy: Exact Risk of M-Estimators under Infinite-Variance Noise in High Dimensions](https://mlsanity.com/paper/2603.28359/) (2026)
- [Influence Diagnostics in High-dimensional M-estimation: Precise Asymptotics](https://mlsanity.com/paper/2607.09250/) (2026)

## Abstract (arXiv)

We investigate the high-dimensional properties of robust regression estimators in the presence of heavy-tailed contamination of both the covariates and response functions. In particular, we provide a sharp asymptotic characterisation of M-estimators trained on a family of elliptical covariate and noise data distributions including cases where second and higher moments do not exist. We show that, despite being consistent, the Huber loss with optimally tuned location parameter δ is suboptimal in the high-dimensional regime in the presence of heavy-tailed noise, highlighting the necessity of further regularisation to achieve optimal performance. This result also uncovers the existence of a transition in δ as a function of the sample complexity and contamination. Moreover, we derive the decay rates for the excess risk of ridge regression. We show that, while it is both optimal and universal for covariate distributions with finite second moment, its decay rate can be considerably faster when the covariates' second moment does not exist. Finally, we show that our formulas readily generalise to a richer family of models and data distributions, such as generalised linear estimation with arbitrary convex regularisation trained on mixture models.