# On the Self-Penalization Phenomenon in Feature Selection

arXiv: https://arxiv.org/abs/2110.05852 · HTML: https://mlsanity.com/paper/2110.05852/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 7 of 7)

- [Characterizing Implicit Bias in Terms of Optimization Geometry](https://mlsanity.com/paper/1802.08246/) (2018)
- [Stochastic Methods for Composite and Weakly Convex Optimization Problems](https://mlsanity.com/paper/1703.08570/) (2017)
- [A simple measure of conditional dependence](https://mlsanity.com/paper/1910.12327/) (2019)
- [Margins, Shrinkage, and Boosting](https://mlsanity.com/paper/1303.4172/) (2013)
- [A New Perspective on Boosting in Linear Regression via Subgradient Optimization and Relatives](https://mlsanity.com/paper/1505.04243/) (2015)
- [Taming Nonconvexity in Kernel Feature Selection -- Favorable Properties of the Laplace Kernel](https://mlsanity.com/paper/2106.09387/) (2021)
- [A Self-Penalizing Objective Function for Scalable Interaction Detection](https://mlsanity.com/paper/2011.12215/) (2020)

## Built on by (most cited first, 4 of 4)

- [Gradient Norm Minimization of Nesterov Acceleration: o(1/k3)](https://mlsanity.com/paper/2209.08862/) (2022)
- [Scalable variable selection for two-view learning tasks with projection operators](https://mlsanity.com/paper/2307.01558/) (2023)
- [A Compositional Kernel Model for Feature Learning](https://mlsanity.com/paper/2509.14158/) (2025)
- [RFFNet: Large-Scale Interpretable Kernel Methods via Random Fourier Features](https://mlsanity.com/paper/2211.06410/) (2022)

## Abstract (arXiv)

We describe an implicit sparsity-inducing mechanism based on minimization over a family of kernels: beginequation* minβ, f~widehatmathbbE[L(Y, f(β1/q odot X)] + λn |f|Hq2~~subject to~~β ≥ 0, endequation* where L is the loss, odot is coordinate-wise multiplication and Hq is the reproducing kernel Hilbert space based on the kernel kq(x, x') = h(|x-x'|qq), where |·|q is the ℓq norm. Using gradient descent to optimize this objective with respect to β leads to exactly sparse stationary points with high probability. The sparsity is achieved without using any of the well-known explicit sparsification techniques such as penalization (e.g., ℓ1), early stopping or post-processing (e.g., clipping). As an application, we use this sparsity-inducing mechanism to build algorithms consistent for feature selection.