Data to 7 Oct 2026

Soft Weight-Sharing for Neural Network Compression

arXiv 1702.04008 ↗First posted 13 Feb 2017stat.ML cs.LGcited by 99

Not on the map. Why?

Abstract (arXiv)

The success of deep learning in numerous application domains created the de- sire to run and train them on mobile devices. This however, conflicts with their computationally, memory and energy intense nature, leading to a growing interest in compression. Recent work by Han et al. (2015a) propose a pipeline that involves retraining, pruning and quantization of neural network weights, obtaining state-of-the-art compression rates. In this paper, we show that competitive compression rates can be achieved by using a version of soft weight-sharing (Nowlan & Hinton, 1992). Our method achieves both quantization and pruning in one simple (re-)training procedure. This point of view also exposes the relation between compression and the minimum description length (MDL) principle.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 22; 2023: 20; 2024: 8; 2025: 19; 2026: 6.

99 papers.

Sort:

By area:

2021COIN: COmpression with Implicit Neural representationscited by 232
2021CDFI: Compression-Driven Network Design for Frame Interpolationcited by 55
2021Differentiable Model Compression via Pseudo Quantization Noisecited by 44
2025Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Modelscited by 44
2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33
2021DKM: Differentiable K-Means Clustering Layer for Neural Network Compressioncited by 31
2022Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraintscited by 25
2021An Information-Theoretic Justification for Model Pruningcited by 23
2021Bit-Mixer: Mixed-precision networks with runtime bit-width selectioncited by 20
2022Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptionscited by 20
2022Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNscited by 16
2021HyperSPNs: Compact and Expressive Probabilistic Circuitscited by 14

38 more

2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 14
2024GenAINet: Enabling Wireless Collective Intelligence via Knowledge Transfer and Reasoningcited by 13
2024Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Modelscited by 13
2021SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networkscited by 12
2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12
2022Bottlenecks CLUB: Unifying Information-Theoretic Trade-offs Among Complexity, Leakage, and Utilitycited by 12
2023Geometry-aware training of factorized layers in tensor Tucker formatcited by 12
2021Sparse PointPillars: Maintaining and Exploiting Input Sparsity to Improve Runtime on Embedded Systemscited by 11
2021Neural network relief: a pruning algorithm based on neural activitycited by 10
2021Finding the Task-Optimal Low-Bit Sub-Distribution in Deep Neural Networkscited by 9
2023Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Reviewcited by 9
2024FoldGPT: Simple and Effective Large Language Model Compression Schemecited by 9
2023Quantum HyperNetworks: Training Binary Neural Networks in Quantum Superpositioncited by 8
2024Parametric PDE Control with Deep Reinforcement Learning and Differentiable L0-Sparse Polynomial Policiescited by 7
2021A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statisticscited by 6
2022REx: Data-Free Residual Quantization Error Expansioncited by 6
2022Sparsity-guided Network Design for Frame Interpolationcited by 6
2025Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformerscited by 6
2021Distilling Image Classifiers in Object Detectorscited by 5
2022Reducing Redundancy in the Bottleneck Representation of the Autoencoderscited by 5
2022Physics-aware Differentiable Discrete Codesign for Diffractive Optical Neural Networkscited by 5
2023Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Trainingcited by 5
2021Lossless Compression with Latent Variable Modelscited by 4
2021HEMP: High-order Entropy Minimization for neural network comPressioncited by 4
2022Automatic Sparse Connectivity Learning for Neural Networkscited by 4
2023Lossy and Lossless (L2) Post-training Model Size Compressioncited by 4
2023eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Modelscited by 4
2024Decoupled Weight Decay for Any p Normcited by 4
2024MCNC: Manifold-Constrained Reparameterization for Neural Compressioncited by 4
2021Honey, I Shrunk The Actor: A Case Study on Preserving Performance with Smaller Actors in Actor-Critic RLcited by 3
2022Regularization-based Pruning of Irrelevant Weights in Deep Neural Architecturescited by 3
2022AMED: Automatic Mixed-Precision Quantization for Edge Devicescited by 3
2023Efficient and Effective Methods for Mixed Precision Neural Network Quantization for Faster, Energy-efficient Inferencecited by 3
2023Understanding Parameter Sharing in Transformerscited by 3
2021A Deeper Look into Convolutions via Eigenvalue-based Pruningcited by 2
2023Weight-Sharing Regularizationcited by 2
2025Pruning-Based TinyML Optimization of Machine Learning Models for Anomaly Detection in Electric Vehicle Charging Infrastructurecited by 2
2025Stochastic Weight Sharing for Bayesian Neural Networkscited by 2

50 most cited of 99 All 99 (JSON)