mlsanityData to 7 Oct 2026

Robust Learning of a Group DRO Neuron

arXiv 2601.18115 ↗First posted 26 Jan 2026cs.LG cs.DS math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

We study the problem of learning a single neuron under standard squared loss in the presence of arbitrary label noise and group-level distributional shifts, for a broad family of covariate distributions. Our goal is to identify a ''best-fit'' neuron parameterized by w_* that performs well under the most challenging reweighting of the groups. Specifically, we address a Group Distributionally Robust Optimization problem: given sample access to K distinct distributions mathcal p[1],dots,mathcal p[K], we seek to approximate w_* that minimizes the worst-case objective over convex combinations of group distributions boldsymbolλ ∈ DeltaK, where the objective is sumi ∈ [K]λ[i],mathbb E_(mathbf x,y)~mathcal p[i](σ(mathbf w·mathbf x)-y)2 - nu df(boldsymbolλ,frac1Kmathbf1) and df is an f-divergence that imposes (optional) penalty on deviations from uniform group weights, scaled by a parameter nu ≥ 0. We develop a computationally efficient primal-dual algorithm that outputs a vector widehatmathbf w that is constant-factor competitive with w_* under the worst-case group weighting. Our analytical framework directly confronts the inherent nonconvexity of the loss function, providing robust learning guarantees in the face of arbitrary label corruptions and group-specific distributional shifts. The implementation of the dual extrapolation update motivated by our algorithmic framework shows promise on LLM pre-training benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 60 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

34 of 60 references.

References · 34

  1. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  2. 2018A Survey on Deep Transfer Learningcited by 417
  3. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  4. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  5. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  6. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  7. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  8. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 266
  9. 2019Distributionally Robust Language Modelingcited by 142
  10. 2018Weakly-Convex Concave Min-Max Optimization: Provable Algorithms and Applications in Machine Learningcited by 62
  11. 2020Near-Optimal SQ Lower Bounds for Agnostically Learning Halfspaces and ReLUs under Gaussian Marginalscited by 57
  12. 2020A Single-Loop Smoothed Gradient Descent-Ascent Algorithm for Nonconvex-Concave Min-Max Problemscited by 57
22 more
  1. 2017Learning ReLUs via Gradient Descentcited by 54
  2. 2019Distribution-Independent PAC Learning of Halfspaces with Massart Noisecited by 54
  3. 2021The Optimality of Polynomial Regression for Agnostic Learning under Gaussian Marginalscited by 52
  4. 2020Learning a Single Neuron with Gradient Methodscited by 50
  5. 2020Statistical-Query Lower Bounds via Functional Gradientscited by 50
  6. 2020An Online Method for A Class of Distributionally Robust Optimization with Non-Convex Objectivescited by 42
  7. 2022On-Demand Sampling: Learning Optimally from Multiple Distributionscited by 39
  8. 2020Simple and optimal methods for stochastic variational inequalities, I: operator extrapolationcited by 35
  9. 2020Approximation Schemes for ReLU Regressioncited by 34
  10. 2019Time/Accuracy Tradeoffs for Learning a ReLU with respect to Gaussian Marginalscited by 32
  11. 2022Minimax Regret Optimization for Robust Machine Learning under Distribution Shiftcited by 31
  12. 2023Near-Optimal Cryptographic Hardness of Agnostically Learning Halfspaces and ReLU Regression under Gaussian Marginalscited by 31
  13. 2020Class-Weighted Classification: Trade-offs and Robust Approachescited by 30
  14. 2021Statistical Analysis of Wasserstein Distributionally Robust Estimatorscited by 27
  15. 2021No-Regret Dynamics in the Fenchel Game: A Unified Framework for Algorithmic Convex Optimizationcited by 20
  16. 2022Learning a Single Neuron with Adversarial Label Noise via Gradient Descentcited by 20
  17. 2022Nonsmooth Nonconvex-Nonconcave Minimax Optimization: Primal-Dual Balancing and Iteration Complexity Analysiscited by 12
  18. 2023Robustly Learning a Single Neuron via Sharpnesscited by 12
  19. 2019Fitting ReLUs via SGD and Quantized SGDcited by 10
  20. 2021ReLU Regression with Massart Noisecited by 10
  21. 2024Learning a Single Neuron Robustly to Distributional Shifts and Adversarial Label Noisecited by 2
  22. 2025Efficient Methods for Min-Max Optimization with Dual-Linear Couplingcited by 1

Built on by

Not cited yet.