mlsanityData to 7 Oct 2026

IDKM: Memory Efficient Neural Network Quantization via Implicit, Differentiable k-Means

arXiv 2312.07759 ↗First posted 12 Dec 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Compressing large neural networks with minimal performance loss is crucial to enabling their deployment on edge devices. (Cho et al., 2022) proposed a weight quantization method that uses an attention-based clustering algorithm called differentiable k-means (DKM). Despite achieving state-of-the-art results, DKM's performance is constrained by its heavy memory dependency. We propose an implicit, differentiable k-means algorithm (IDKM), which eliminates the major memory restriction of DKM. Let t be the number of k-means iterations, m be the number of weight-vectors, and b be the number of bits per cluster address. IDKM reduces the overall memory complexity of a single k-means layer from O(t · m · 2b) to O( m · 2b). We also introduce a variant, IDKM with Jacobian-Free-Backpropagation (IDKM-JFB), for which the time complexity of the gradient calculation is independent of t as well. We provide a proof of concept of our methods by showing that, under the same settings, IDKM achieves comparable performance to DKM with less compute time and less memory. We also use IDKM and IDKM-JFB to quantize a large neural network, Resnet18, on hardware where DKM cannot train at all.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

24 of 31 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  8. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  9. 2019Deep Equilibrium Modelscited by 603
  10. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  11. 2017Towards Accurate Binary Convolutional Neural Networkcited by 148
  12. 2019Implicit Deep Learningcited by 124
12 more
  1. 2020Monotone operator equilibrium networkscited by 100
  2. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  3. 2021JFB: Jacobian-Free Backpropagation for Implicit Networkscited by 84
  4. 2019And the Bit Goes Down: Revisiting the Quantization of Neural Networkscited by 78
  5. 2020Post-Training Piecewise Linear Quantization for Deep Neural Networkscited by 77
  6. 2020An Overview of Neural Network Compressioncited by 59
  7. 2020PROFIT: A Novel Training Method for sub-4-bit MobileNet Modelscited by 50
  8. 2020Methods for Pruning Deep Neural Networkscited by 49
  9. 2018Recurrent Stacking of Layers for Compact Neural Machine Translation Modelscited by 38
  10. 2018Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutionscited by 33
  11. 2021DKM: Differentiable K-Means Clustering Layer for Neural Network Compressioncited by 31
  12. 2021Robust Implicit Networks via Non-Euclidean Contractionscited by 19

Built on by

Not cited yet.