# IHT dies hard: Provable accelerated Iterative Hard Thresholding

arXiv: https://arxiv.org/abs/1712.09379 · HTML: https://mlsanity.com/paper/1712.09379/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Dec 2017.

## Built on by (most cited first, 10 of 10)

- [Fast quantum state reconstruction via accelerated non-convex programming](https://mlsanity.com/paper/2104.07006/) (2021)
- [One Rank at a Time: Cascading Error Dynamics in Sequential Learning](https://mlsanity.com/paper/2505.22602/) (2025)
- [On Asymptotic Linear Convergence of Projected Gradient Descent for Constrained Least Squares](https://mlsanity.com/paper/2112.11760/) (2021)
- [Convergence of the mini-batch SIHT algorithm](https://mlsanity.com/paper/2209.14536/) (2022)
- [Sparse Polyak: an adaptive step size rule for high-dimensional M-estimation](https://mlsanity.com/paper/2509.09802/) (2025)
- [i-SpaSP: Structured Neural Pruning via Sparse Signal Recovery](https://mlsanity.com/paper/2112.04905/) (2021)
- [Batch Active Learning from the Perspective of Sparse Approximation](https://mlsanity.com/paper/2211.00246/) (2022)
- [Error-Correction for Sparse Support Recovery Algorithms](https://mlsanity.com/paper/2103.03801/) (2021)
- [A Recovery Guarantee for Sparse Neural Networks](https://mlsanity.com/paper/2509.20323/) (2025)
- [Sparse Polyak with optimal thresholding operators for high-dimensional M-estimation](https://mlsanity.com/paper/2511.18167/) (2025)

## Abstract (arXiv)

We study --both in theory and practice-- the use of momentum motions in classic iterative hard thresholding (IHT) methods. By simply modifying plain IHT, we investigate its convergence behavior on convex optimization criteria with non-convex constraints, under standard assumptions. In diverse scenaria, we observe that acceleration in IHT leads to significant improvements, compared to state of the art projected gradient descent and Frank-Wolfe variants. As a byproduct of our inspection, we study the impact of selecting the momentum parameter: similar to convex settings, two modes of behavior are observed --"rippling" and linear-- depending on the level of momentum.