# High-Dimensional Linear Regression via Implicit Regularization

arXiv: https://arxiv.org/abs/1903.09367 · HTML: https://mlsanity.com/paper/1903.09367/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2019.

## Built on by (most cited first, 25 of 25)

- [Mask in the Mirror: Implicit Sparsification](https://mlsanity.com/paper/2408.09966/) (2024)
- [Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries](https://mlsanity.com/paper/2502.02496/) (2025)
- [When does Privileged Information Explain Away Label Noise?](https://mlsanity.com/paper/2303.01806/) (2023)
- [Sequential Attention for Feature Selection](https://mlsanity.com/paper/2209.14881/) (2022)
- [Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch](https://mlsanity.com/paper/2504.12801/) (2025)
- [Nearly Minimax-Optimal Rates for Noisy Sparse Phase Retrieval via Early-Stopped Mirror Descent](https://mlsanity.com/paper/2105.03678/) (2021)
- [Computationally Efficient and Statistically Optimal Robust High-Dimensional Linear Regression](https://mlsanity.com/paper/2305.06199/) (2023)
- [Implicit regularization in AI meets generalized hardness of approximation in optimization -- Sharp results for diagonal linear networks](https://mlsanity.com/paper/2307.07410/) (2023)
- [FTMixer: Frequency and Time Domain Representations Fusion for Time Series Modeling](https://mlsanity.com/paper/2405.15256/) (2024)
- [Diagonal Linear Networks and the Lasso Regularization Path](https://mlsanity.com/paper/2509.18766/) (2025)
- [Never Saddle for Reparameterized Steepest Descent as Mirror Flow](https://mlsanity.com/paper/2603.02064/) (2026)
- [Improving Adaptivity via Over-Parameterization in Sequence Models](https://mlsanity.com/paper/2409.00894/) (2024)
- [Differentiable Sparsity via D-Gating: Simple and Versatile Structured Penalization](https://mlsanity.com/paper/2509.23898/) (2025)
- [Precise asymptotics of reweighted least-squares algorithms for linear diagonal networks](https://mlsanity.com/paper/2406.02769/) (2024)
- [Optimization Insights into Deep Diagonal Linear Networks](https://mlsanity.com/paper/2412.16765/) (2024)
- [Acceleration and Implicit Regularization in Gaussian Phase Retrieval](https://mlsanity.com/paper/2311.12888/) (2023)
- [Simultaneous Best Subset Selection and Dimension Reduction via Primal-Dual Iterations](https://mlsanity.com/paper/2211.15889/) (2022)
- [Iterative Regularization with k-support Norm: An Important Complement to Sparse Recovery](https://mlsanity.com/paper/2401.05394/) (2023)
- [Implicit Regularization for Multi-label Feature Selection](https://mlsanity.com/paper/2411.11436/) (2024)
- [Towards a Statistical Understanding of Neural Networks: Beyond the Neural Tangent Kernel Theories](https://mlsanity.com/paper/2412.18756/) (2024)
- [Leveraging Label Potential for Enhanced Multimodal Emotion Recognition](https://mlsanity.com/paper/2504.05158/) (2025)
- [Tuning-Free Structured Sparse Recovery of Multiple Measurement Vectors using Implicit Regularization](https://mlsanity.com/paper/2512.03393/) (2025)
- [HORST: Composing Optimizer Geometries for Sparse Transformer Training](https://mlsanity.com/paper/2605.21104/) (2026)
- [Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate](https://mlsanity.com/paper/2605.25134/) (2026)
- [Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks](https://mlsanity.com/paper/2609.34756/) (2026)

## Abstract (arXiv)

Many statistical estimators for high-dimensional linear regression are M-estimators, formed through minimizing a data-dependent square loss function plus a regularizer. This work considers a new class of estimators implicitly defined through a discretized gradient dynamic system under overparameterization. We show that under suitable restricted isometry conditions, overparameterization leads to implicit regularization: if we directly apply gradient descent to the residual sum of squares with sufficiently small initial values, then under some proper early stopping rule, the iterates converge to a nearly sparse rate-optimal solution that improves over explicitly regularized approaches. In particular, the resulting estimator does not suffer from extra bias due to explicit penalties, and can achieve the parametric root-n rate when the signal-to-noise ratio is sufficiently high. We also perform simulations to compare our methods with high dimensional linear regression with explicit regularization. Our results illustrate the advantages of using implicit regularization via gradient descent after overparameterization in sparse vector estimation.