# Learned Threshold Pruning

arXiv: https://arxiv.org/abs/2003.00075 · HTML: https://mlsanity.com/paper/2003.00075/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2020.

## Built on by (most cited first, 17 of 17)

- [What Matters In The Structured Pruning of Generative Language Models?](https://mlsanity.com/paper/2302.03773/) (2023)
- [Understanding the Dynamics of DNNs Using Graph Modularity](https://mlsanity.com/paper/2111.12485/) (2021)
- [An Expectation-Maximization Perspective on Federated Learning](https://mlsanity.com/paper/2111.10192/) (2021)
- [Accelerating Attention through Gradient-Based Learned Runtime Pruning](https://mlsanity.com/paper/2204.03227/) (2022)
- [Spartan: Differentiable Sparsity via Regularized Transportation](https://mlsanity.com/paper/2205.14107/) (2022)
- [Cyclical Pruning for Sparse Neural Networks](https://mlsanity.com/paper/2202.01290/) (2022)
- [Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection](https://mlsanity.com/paper/2310.13183/) (2023)
- [SD-Conv: Towards the Parameter-Efficiency of Dynamic Convolution](https://mlsanity.com/paper/2204.02227/) (2022)
- [DASS: Differentiable Architecture Search for Sparse neural networks](https://mlsanity.com/paper/2207.06968/) (2022)
- [AUTOSPARSE: Towards Automated Sparse Training of Deep Neural Networks](https://mlsanity.com/paper/2304.06941/) (2023)
- [Learning Pruned Structure and Weights Simultaneously from Scratch: an Attention based Approach](https://mlsanity.com/paper/2111.02399/) (2021)
- [Learning Activation Functions for Sparse Neural Networks](https://mlsanity.com/paper/2305.10964/) (2023)
- [A Novel Sparse Regularizer](https://mlsanity.com/paper/2301.07285/) (2023)
- [What to Do When Your Discrete Optimization Is the Size of a Neural Network?](https://mlsanity.com/paper/2402.10339/) (2024)
- [UnIT: Scalable Unstructured Inference-Time Pruning for MAC-efficient Neural Inference on MCUs](https://mlsanity.com/paper/2507.07885/) (2025)
- [Robust and Interpretable Adaptation of Equivariant Materials Foundation Models via Sparsity-promoting Fine-tuning](https://mlsanity.com/paper/2606.18691/) (2026)
- [Your Transformer Is Secretly Elastic](https://mlsanity.com/paper/2610.20588/) (2026)

## Abstract (arXiv)

This paper presents a novel differentiable method for unstructured weight pruning of deep neural networks. Our learned-threshold pruning (LTP) method learns per-layer thresholds via gradient descent, unlike conventional methods where they are set as input. Making thresholds trainable also makes LTP computationally efficient, hence scalable to deeper networks. For example, it takes 30 epochs for LTP to prune ResNet50 on ImageNet by a factor of 9.1. This is in contrast to other methods that search for per-layer thresholds via a computationally intensive iterative pruning and fine-tuning process. Additionally, with a novel differentiable L0 regularization, LTP is able to operate effectively on architectures with batch-normalization. This is important since L1 and L2 penalties lose their regularizing effect in networks with batch-normalization. Finally, LTP generates a trail of progressively sparser networks from which the desired pruned network can be picked based on sparsity and performance requirements. These features allow LTP to achieve competitive compression rates on ImageNet networks such as AlexNet (26.4× compression with 79.1% Top-5 accuracy) and ResNet50 (9.1× compression with 92.0% Top-5 accuracy). We also show that LTP effectively prunes modern textitcompact architectures, such as EfficientNet, MobileNetV2 and MixNet.