mlsanityData to 7 Oct 2026

DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures

arXiv 1908.09979 ↗First posted 27 Aug 2019cs.LG stat.MLcited by 56

Not on the map. Why?

Abstract (arXiv)

In seeking for sparse and efficient neural network models, many previous works investigated on enforcing L1 or L0 regularizers to encourage weight sparsity during training. The L0 regularizer measures the parameter sparsity directly and is invariant to the scaling of parameter values, but it cannot provide useful gradients, and therefore requires complex optimization techniques. The L1 regularizer is almost everywhere differentiable and can be easily optimized with gradient descent. Yet it is not scale-invariant, causing the same shrinking rate to all parameters, which is inefficient in increasing sparsity. Inspired by the Hoyer measure (the ratio between L1 and L2 norms) used in traditional compressed sensing problems, we present DeepHoyer, a set of sparsity-inducing regularizers that are both differentiable almost everywhere and scale-invariant. Our experiments show that enforcing DeepHoyer regularizers can produce even sparser neural network models than previous works, under the same accuracy level. We also show that DeepHoyer can be applied to both element-wise and structural pruning.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 11; 2023: 11; 2024: 9; 2025: 9; 2026: 5.

56 papers.

Sort:

By area:

  1. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  2. 2021How Well Do Sparse Imagenet Models Transfer?cited by 50
  3. 2021BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantizationcited by 37
  4. 2023OTOV2: Automatic, Generic, User-Friendlycited by 32
  5. 2021ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximationscited by 21
  6. 2021Single-Shot Pruning for Offline Reinforcement Learningcited by 21
  7. 2021Neural Network Training Using ℓ1-Regularization and Bi-fidelity Datacited by 16
  8. 2022Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNscited by 16
  9. 2022Trainability Preserving Neural Pruningcited by 16
  10. 2022Efficient Stein Variational Inference for Reliable Distribution-lossless Network Pruningcited by 16
  11. 2025Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compressioncited by 14
  12. 2021Do All MobileNets Quantize Poorly? Gaining Insights into the Effect of Quantization on Depthwise Separable Convolutional Networks Through the Eyes of Multi-scale Distributional Dynamicscited by 12
38 more
  1. 2021Towards Structured Dynamic Sparse Pre-Training of BERTcited by 12
  2. 2023Automatic Attention Pruning: Improving and Automating Model Pruning using Attentionscited by 11
  3. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  4. 2021Model Preserving Compression for Neural Networkscited by 9
  5. 2022On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networkscited by 9
  6. 2024ProtoS-ViT: Visual foundation models for sparse self-explainable classificationscited by 8
  7. 2023OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operatorscited by 6
  8. 2023Dynamic Structure Pruning for Compressing CNNscited by 5
  9. 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruningcited by 5
  10. 2022Optimizing the Consumption of Spiking Neural Networks with Activity Regularizationcited by 4
  11. 2022Hoyer regularizer is all you need for ultra low-latency spiking neural networkscited by 4
  12. 2024HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruningcited by 4
  13. 2021Variational Nested Dropoutcited by 3
  14. 2022Adaptive Activation-based Structured Pruningcited by 3
  15. 2022Nonlinear Initialization Methods for Low-Rank Neural Networkscited by 3
  16. 2022Receding Neuron Importances for Structured Pruningcited by 3
  17. 2023Pruning Before Training May Improve Generalization, Provablycited by 3
  18. 2024Enhanced Sparsification via Stimulative Trainingcited by 2
  19. 2025Tell me why: Visual foundation models as self-explainable classifierscited by 2
  20. 2025Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsitycited by 2
  21. 2021BWCP: Probabilistic Learning-to-Prune Channels for ConvNets via Batch Whiteningcited by 1
  22. 2022Optimal channel selection with discrete QCQPcited by 1
  23. 2022Learning k-Level Structured Sparse Neural Networks Using Group Envelope Regularizationcited by 1
  24. 2023Sparse and Local Networks for Hypergraph Reasoningcited by 1
  25. 2023SalientGrads: Sparse Models for Communication Efficient and Data Aware Distributed Federated Trainingcited by 1
  26. 2023End-to-End Neural Network Compression via fracℓ1ℓ2 Regularized Latency Surrogatescited by 1
  27. 2024Always-Sparse Training by Growing Connections with Guided Stochastic Explorationcited by 1
  28. 2024SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learningcited by 1
  29. 2024Decay Pruning Method: Smooth Pruning With a Self-Rectifying Procedurecited by 1
  30. 2024Fisher-aware Quantization for DETR Detectors with Critical-category Objectivescited by 1
  31. 2026Extracting Governing Equations from Latent Dynamics via Multi-View Contrastive Learningcited by 1
  32. 2026SLORR: Simple and Efficient In-Training Low-Rank Regularizationcited by 1
  33. 2023A Novel Sparse Regularizercited by 0
  34. 2023HCE: Improving Performance and Efficiency with Heterogeneously Compressed Neural Network Ensemblecited by 0
  35. 2023A Tensor-based Convolutional Neural Network for Small Dataset Classificationcited by 0
  36. 2024LNPT: Label-free Network Pruning and Trainingcited by 0
  37. 2024Pruning for Improved ADC Efficiency in Crossbar-based Analog In-memory Acceleratorscited by 0
  38. 2025Morphing-based Compression for Data-centric ML Pipelinescited by 0

50 most cited of 56 All 56 (JSON)