Data to 14 Oct 2026

Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy

arXiv 1711.05852 ↗First posted 15 Nov 2017cs.LG cs.CV cs.NEcited by 101

Not on the map. Why?

Cited by 27 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous parameters. Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems - the models (often deep networks or wide networks or both) are compute and memory intensive. Low-precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footprint of these deployed models. In this paper, we study the combination of these two techniques and show that the performance of low-precision networks can be significantly improved by using knowledge distillation techniques. Our approach, Apprentice, achieves state-of-the-art accuracies using ternary precision and 4-bit precision for variants of ResNet architecture on ImageNet dataset. We present three schemes using which one can apply knowledge distillation techniques to various stages of the train-and-deploy pipeline.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 37; 2022: 19; 2023: 21; 2024: 11; 2025: 9; 2026: 4.

101 papers.

Sort:

By area:

2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
2021Knowledge distillation: A good teacher is patient and consistentcited by 259
2021I-BERT: Integer-only BERT Quantizationcited by 255
2021Does Knowledge Distillation Really Work?cited by 190
2021SEED: Self-supervised Distillation For Visual Representationcited by 134
2021Network Quantization with Element-wise Gradient Scalingcited by 87
2022Unified Visual Transformer Compressioncited by 79
2022Knowledge Distillation via the Target-aware Transformercited by 70
2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 65
2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 64
2022MAViL: Masked Audio-Video Learnerscited by 59
2021Hessian-Aware Pruning and Optimal Neural Implantcited by 45

38 more

2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
2021Extremely Lightweight Quantization Robust Real-Time Single-Image Super Resolution for Mobile Devicescited by 30
2021Auto-Split: A General Framework of Collaborative Edge-Cloud AIcited by 30
2022BiViT: Extremely Compressed Binary Vision Transformercited by 28
2023Multimodal Distillation for Egocentric Action Recognitioncited by 28
2024CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Modelscited by 26
2022Revisiting Self-Distillationcited by 24
2022Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networkscited by 23
2021Bit-Mixer: Mixed-precision networks with runtime bit-width selectioncited by 22
2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 22
2022EcoFormer: Energy-Saving Attention with Linear Complexitycited by 21
2023SlimSAM: 0.1% Data Makes Segment Anything Slimcited by 20
2021CPT: Efficient Deep Neural Network Training via Cyclic Precisioncited by 19
2021"BNN - BN = ?": Training Binary Neural Networks without Batch Normalizationcited by 17
2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
2023Quantized Feature Distillation for Network Quantizationcited by 15
2021Neural Network Quantization for Efficient Inference: A Surveycited by 14
2023A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Servicescited by 14
2021CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognitioncited by 11
2021LANA: Latency Aware Network Accelerationcited by 11
2021Towards Data-Free Domain Generalizationcited by 11
2022PCA-Based Knowledge Distillation Towards Lightweight and Content-Style Balanced Photorealistic Style Transfer Modelscited by 11
2023Knowledge Distillation Performs Partial Variance Reductioncited by 11
2021Finding the Task-Optimal Low-Bit Sub-Distribution in Deep Neural Networkscited by 9
2021Knowledge Distillation via Instance-level Sequence Learningcited by 8
2023QD-BEV : Quantization-aware View-guided Distillation for Multi-view 3D Object Detectioncited by 8
2024The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Informationcited by 8
2024Applications of Knowledge Distillation in Remote Sensing: A Surveycited by 8
2022Mixed-Precision Neural Networks: A Surveycited by 7
2022Designing and Training of Lightweight Neural Networks on Edge Devices using Early Halting in Knowledge Distillationcited by 7
2021Knowledge Distillation Circumvents Nonlinearity for Optical Convolutional Neural Networkscited by 6
2023EPTQ: Enhanced Post-Training Quantization via Hessian-guided Network-wise Optimizationcited by 6
2025Enhancing Physical Consistency in Lightweight World Modelscited by 5
2021TENT: Efficient Quantization of Neural Networks on the tiny Edge with Tapered FixEd PoiNTcited by 4
2021NullaNet Tiny: Ultra-low-latency DNN Inference Through Fixed-function Combinational Logiccited by 4
2021Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillationcited by 4
2022Overlooked Poses Actually Make Sense: Distilling Privileged Knowledge for Human Motion Predictioncited by 4
2022Neural Networks with Quantization Constraintscited by 4

50 most cited of 101 All 101 (JSON)