# Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization

arXiv: https://arxiv.org/abs/2307.03571 · HTML: https://mlsanity.com/paper/2307.03571/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 38 references that are in the map.
- First posted on arXiv 7 Jul 2023.

## Builds on (most cited first, 10 of 10)

- [Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks](https://mlsanity.com/paper/2306.04251/) (2023)
- [(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability](https://mlsanity.com/paper/2302.08982/) (2023)
- [More is Less: Inducing Sparsity via Overparameterization](https://mlsanity.com/paper/2112.11027/) (2021)
- [The effect of smooth parametrizations on nonconvex optimization landscapes](https://mlsanity.com/paper/2207.03512/) (2022)
- [Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networks](https://mlsanity.com/paper/2110.09548/) (2021)
- [Smooth over-parameterized solvers for non-smooth structured optimization](https://mlsanity.com/paper/2205.01385/) (2022)
- [Learning Deep Models: Critical Points and Local Openness](https://mlsanity.com/paper/1803.02968/) (2018)
- [A critical review of LASSO and its derivatives for variable selection under dependence among covariates](https://mlsanity.com/paper/2012.11470/) (2020)
- [Perspective Functions: Proximal Calculus and Applications in High-Dimensional Statistics](https://mlsanity.com/paper/1610.01478/) (2016)
- [Neuronized Priors for Bayesian Sparse Linear Regression](https://mlsanity.com/paper/1810.00141/) (2018)

## Built on by (most cited first, 11 of 11)

- [Q-DiT4SR: Exploration of Detail-Preserving Diffusion Transformer Quantization for Real-World Image Super-Resolution](https://mlsanity.com/paper/2602.01273/) (2026)
- [On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2603.22030/) (2026)
- [Online reinforcement learning via sparse Gaussian mixture model Q-functions](https://mlsanity.com/paper/2509.14585/) (2025)
- [On the Epistemic Uncertainty of Overparametrized Neural Networks](https://mlsanity.com/paper/2605.25234/) (2026)
- [WEEP: A Differentiable Nonconvex Sparse Regularizer via Weakly-Convex Envelope](https://mlsanity.com/paper/2507.20447/) (2025)
- [Global Minimizers of ℓp-Regularized Objectives Yield the Sparsest ReLU Neural Networks](https://mlsanity.com/paper/2505.21791/) (2025)
- [Kernel Regression of Multi-Way Data via Tensor Trains with Hadamard Overparametrization: The Dynamic Graph Flow Case](https://mlsanity.com/paper/2509.22197/) (2025)
- [HORST: Composing Optimizer Geometries for Sparse Transformer Training](https://mlsanity.com/paper/2605.21104/) (2026)
- [Deep Unfolded Latent Optimally Partitioned-l2/l1 Networks for Data-driven Block-Sparse Recovery](https://mlsanity.com/paper/2606.12740/) (2026)
- [Kernel Regression with Tensor Trains and Hadamard Overparameterization](https://mlsanity.com/paper/2607.17390/) (2026)
- [Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning](https://mlsanity.com/paper/2607.23474/) (2026)

## Abstract (arXiv)

We present a framework for smooth optimization of explicitly regularized objectives for (structured) sparsity. These non-smooth and possibly non-convex problems typically rely on solvers tailored to specific models and regularizers. In contrast, our method enables fully differentiable and approximation-free optimization and is thus compatible with the ubiquitous gradient descent paradigm in deep learning. The proposed optimization transfer comprises an overparameterization of selected parameters and a change of penalties. In the overparametrized problem, smooth surrogate regularization induces non-smooth, sparse regularization in the base parametrization. We prove that the surrogate objective is equivalent in the sense that it not only has identical global minima but also matching local minima, thereby avoiding the introduction of spurious solutions. Additionally, our theory establishes results of independent interest regarding matching local minima for arbitrary, potentially unregularized, objectives. We comprehensively review sparsity-inducing parametrizations across different fields that are covered by our general theory, extend their scope, and propose improvements in several aspects. Numerical experiments further demonstrate the correctness and effectiveness of our approach on several sparse learning problems ranging from high-dimensional regression to sparse neural network training.