mlsanityData to 7 Oct 2026

Model compression via distillation and quantization

arXiv 1802.05668 ↗First posted 15 Feb 2018cs.NE cs.LGcited by 287

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning. One aspect of the field receiving considerable attention is efficiently executing deep models in resource-constrained environments, such as mobile or embedded devices. This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger teacher networks into smaller student networks. The first method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher, into the training of a student network whose weights are quantized to a limited set of levels. The second method, differentiable quantization, optimizes the location of quantization points through stochastic gradient descent, to better fit the behavior of the teacher model. We validate both methods through experiments on convolutional and recurrent architectures. We show that quantized shallow students can reach similar accuracy levels to full-precision teacher models, while providing order of magnitude compression, and inference speedup that is linear in the depth reduction. In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.

Key facts. Cited by 287 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 80; 2022: 47; 2023: 54; 2024: 53; 2025: 41; 2026: 12.

287 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  3. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  4. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  5. 2021NeRV: Neural Representations for Videoscited by 255
  6. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  7. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  8. 2021Towards Understanding Knowledge Distillationcited by 158
  9. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  10. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  11. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  12. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
38 more
  1. 2023CompGS: Smaller and Faster Gaussian Splatting with Vector Quantizationcited by 127
  2. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  3. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
  4. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  5. 2021Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyondcited by 75
  6. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  7. 2022Compressing Volumetric Radiance Fields to 1 MBcited by 66
  8. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 64
  9. 2021Diversifying Sample Generation for Accurate Data-Free Quantizationcited by 63
  10. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 59
  11. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
  12. 2021Privacy-Preserving Machine Learning: Methods, Challenges and Directionscited by 58
  13. 2021CDFI: Compression-Driven Network Design for Frame Interpolationcited by 55
  14. 2021A Survey on Green Deep Learningcited by 52
  15. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  16. 2023LightTS: Lightweight Time Series Classification with Adaptive Ensemble Distillation -- Extended Versioncited by 49
  17. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 44
  18. 2021Hessian-Aware Pruning and Optimal Neural Implantcited by 43
  19. 2024Fast Vocabulary Transfer for Language Model Compressioncited by 42
  20. 2021Learnable Companding Quantization for Accurate Low-bit Neural Networkscited by 40
  21. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 38
  22. 2021BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantizationcited by 37
  23. 2023Ternary Spike: Learning Ternary Spikes for Spiking Neural Networkscited by 36
  24. 2022EfficientFi: Towards Large-Scale Lightweight WiFi Sensing via CSI Compressioncited by 35
  25. 2021A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Conceptscited by 33
  26. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
  27. 2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
  28. 2023Artificial General Intelligence for Medical Imaging Analysiscited by 32
  29. 2021DKM: Differentiable K-Means Clustering Layer for Neural Network Compressioncited by 31
  30. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  31. 2023Students Parrot Their Teachers: Membership Inference on Model Distillationcited by 31
  32. 2021Student Network Learning via Evolutionary Knowledge Distillationcited by 30
  33. 2022FedSynth: Gradient Compression via Synthetic Data in Federated Learningcited by 30
  34. 2024How Far Can We Compress Instant-NGP-Based NeRF?cited by 30
  35. 2021Auto-Split: A General Framework of Collaborative Edge-Cloud AIcited by 29
  36. 2021GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarizationcited by 29
  37. 2021ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Trainingcited by 27
  38. 2023RMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networkscited by 27

50 most cited of 287 All 287 (JSON)