mlsanityData to 13 Oct 2026

Partitioning-Guided K-Means: Extreme Empty Cluster Resolution for Extreme Model Compression

arXiv 2306.14031 ↗First posted 24 Jun 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Compactness in deep learning can be critical to a model's viability in low-resource applications, and a common approach to extreme model compression is quantization. We consider Iterative Product Quantization (iPQ) with Quant-Noise to be state-of-the-art in this area, but this quantization framework suffers from preventable inference quality degradation due to prevalent empty clusters. In this paper, we propose several novel enhancements aiming to improve the accuracy of iPQ with Quant-Noise by focusing on resolving empty clusters. Our contribution, which we call Partitioning-Guided k-means (PG k-means), is a heavily augmented k-means implementation composed of three main components. First, we propose a partitioning-based pre-assignment strategy that ensures no initial empty clusters and encourages an even weight-to-cluster distribution. Second, we propose an empirically superior empty cluster resolution heuristic executed via cautious partitioning of large clusters. Finally, we construct an optional optimization step that consolidates intuitively dense clusters of weights to ensure shared representation. The proposed approach consistently reduces the number of empty clusters in iPQ with Quant-Noise by 100x on average, uses 8x fewer iterations during empty cluster resolution, and improves overall model accuracy by up to 12%, when applied to RoBERTa on a variety of tasks in the GLUE benchmark.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 13 of its 20 references that are in the map. First posted on arXiv 24 Jun 2023.

Builds on

13 of 20 references.

References · 13

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,056
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,585
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,487
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,157
  8. 2018Universal Transformerscited by 632
  9. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 589
  10. 2015Deep Learning with Limited Numerical Precisioncited by 372
  11. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  12. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 81
1 more
  1. 2019And the Bit Goes Down: Revisiting the Quantization of Neural Networkscited by 78

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Deep learning model compression using network sensitivity and gradients
  2. 2024SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization
  3. 2026Post Training Quantization for Efficient Dataset Condensation
  4. 20261-Bit Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization
  5. 2022Learning Low-Rank Representations for Model Compression
  6. 2025LCD: Advancing Extreme Low-Bit Clustering for Large Language Models via Knowledge Distillation
  7. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
  8. 2025LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inference