mlsanityData to 7 Oct 2026

DKM: Differentiable K-Means Clustering Layer for Neural Network Compression

arXiv 2108.12659 ↗First posted 28 Aug 2021cs.LG cs.AI cs.CVcited by 31

Not on the map. Why?

Abstract (arXiv)

Deep neural network (DNN) model compression for efficient on-device inference is becoming increasingly important to reduce memory requirements and keep user data on-device. To this end, we propose a novel differentiable k-means clustering layer (DKM) and its application to train-time weight clustering-based DNN model compression. DKM casts k-means clustering as an attention problem and enables joint optimization of the DNN parameters and clustering centroids. Unlike prior works that rely on additional regularizers and parameters, DKM-based compression keeps the original loss function and model architecture fixed. We evaluated DKM-based compression on various DNN models for computer vision and natural language processing (NLP) tasks. Our results demonstrate that DKM delivers superior compression and accuracy trade-off on ImageNet1k and GLUE benchmarks. For example, DKM-based compression can offer 74.5% top-1 ImageNet1k accuracy on ResNet50 DNN model with 3.3MB model size (29.4x model compression factor). For MobileNet-v1, which is a challenging DNN to compress, DKM delivers 63.9% top-1 ImageNet1k accuracy with 0.72 MB model size (22.4x model compression factor). This result is 6.8% higher top-1accuracy and 33% relatively smaller model size than the current state-of-the-art DNN compression algorithms. Additionally, DKM enables compression of DistilBERT model by 11.8x with minimal (1.1%) accuracy loss on GLUE NLP benchmarks.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 41 of its 46 references that are in the map. First posted on arXiv 28 Aug 2021.

Builds on

41 of 46 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  12. 2018DARTS: Differentiable Architecture Searchcited by 1,782
29 more
  1. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  2. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  3. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  4. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  5. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  6. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  7. 2018Model compression via distillation and quantizationcited by 287
  8. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  9. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  10. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  11. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
  12. 2017Towards Accurate Binary Convolutional Neural Networkcited by 148
  13. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  14. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 143
  15. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  16. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  17. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  18. 2018ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Networkcited by 90
  19. 2021Network Quantization with Element-wise Gradient Scalingcited by 82
  20. 2019And the Bit Goes Down: Revisiting the Quantization of Neural Networkscited by 78
  21. 2020Lookahead: A Far-Sighted Alternative of Magnitude-based Pruningcited by 61
  22. 2020PROFIT: A Novel Training Method for sub-4-bit MobileNet Modelscited by 50
  23. 2016Towards the Limit of Network Quantizationcited by 49
  24. 2018Value-aware Quantization for Training and Inference of Neural Networkscited by 49
  25. 2018Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutionscited by 33
  26. 2017Learning Discrete Weights Using the Local Reparameterization Trickcited by 29
  27. 2019TensorFlow Eager: A Multi-Stage, Python-Embedded DSL for Machine Learningcited by 22
  28. 2018Universal Deep Neural Network Compressioncited by 17
  29. 2019Neural Epitome Search for Architecture-Agnostic Network Compressioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 9; 2024: 10; 2025: 4; 2026: 2.

31 papers.

Sort:

By area:

  1. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
  2. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  3. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 53
  4. 2023Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancycited by 44
  5. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  6. 2023Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiencescited by 21
  7. 2024PixOOD: Pixel-Level Out-of-Distribution Detectioncited by 20
  8. 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
  9. 2024A Review of Pseudo-Labeling for Computer Visioncited by 16
  10. 2023PDP: Parameter-free Differentiable Pruning is All You Needcited by 12
  11. 2022Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementationcited by 6
  12. 2024Talaria: Interactively Optimizing Machine Learning Models for Efficient Inferencecited by 6
19 more
  1. 2025LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inferencecited by 6
  2. 2022Hyperspherical Quantization: Toward Smaller and More Accurate Modelscited by 4
  3. 2023eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Modelscited by 4
  4. 2023R2 Loss: Range Restriction Loss for Model Compression and Quantizationcited by 3
  5. 2025ViM-VQ: Efficient Post-Training Vector Quantization for Visual Mambacited by 3
  6. 2022Learning Low-Rank Representations for Model Compressioncited by 2
  7. 2023SPT: Fine-Tuning Transformer-based Language Models Efficiently with Sparsificationcited by 2
  8. 2024Polytopic Autoencoders with Smooth Clustering for Reduced-order Modelling of Flowscited by 2
  9. 2025MetaCluster: Enabling Deep Compression of Kolmogorov-Arnold Networkcited by 2
  10. 2024Separate, Dynamic and Differentiable (SMART) Pruner for Block/Output Channel Pruning on Computer Vision Taskscited by 1
  11. 2022Deep learning model compression using network sensitivity and gradientscited by 0
  12. 2022Hyperspherical Loss-Aware Ternary Quantizationcited by 0
  13. 2023CrAFT: Compression-Aware Fine-Tuning for Efficient Visual Task Adaptationcited by 0
  14. 2023IDKM: Memory Efficient Neural Network Quantization via Implicit, Differentiable k-Meanscited by 0
  15. 2024EncodingNet: A Novel Encoding-based MAC Design for Efficient Neural Network Accelerationcited by 0
  16. 2024MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devicescited by 0
  17. 2025SSVQ: Unleashing the Potential of Vector Quantization with Sign-Splittingcited by 0
  18. 2026Efficient VQ-QAT and Mixed Vector/Linear quantized Neural Networkscited by 0
  19. 2026RAFNet: Region-Aware Fusion Network for Pansharpeningcited by 0