mlsanityData to 7 Oct 2026

BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation

arXiv 2402.10631 ↗First posted 16 Feb 2024cs.CLcited by 80

Not on the map. Why?

Abstract (arXiv)

The upscaling of Large Language Models (LLMs) has yielded impressive advances in natural language processing, yet it also poses significant deployment challenges. Weight quantization has emerged as a widely embraced solution to reduce memory and computational demands. This paper introduces BitDistiller, a framework that synergizes Quantization-Aware Training (QAT) with Knowledge Distillation (KD) to boost the performance of LLMs at ultra-low precisions (sub-4-bit). Specifically, BitDistiller first incorporates a tailored asymmetric quantization and clipping technique to maximally preserve the fidelity of quantized weights, and then proposes a novel Confidence-Aware Kullback-Leibler Divergence (CAKLD) objective, which is employed in a self-distillation manner to enable faster convergence and superior model performance. Empirical evaluations demonstrate that BitDistiller significantly surpasses existing methods in both 3-bit and 2-bit configurations on general language understanding and complex reasoning benchmarks. Notably, BitDistiller is shown to be more cost-effective, demanding fewer data and training resources. The code is available at https://github.com/DD-DuDa/BitDistiller.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

29 of 43 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
17 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  5. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  6. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  7. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  8. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  9. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  10. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 160
  11. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
  12. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  13. 2022FP8 Quantization: The Power of the Exponentcited by 87
  14. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 65
  15. 2022Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Trainingcited by 26
  16. 2023Token-Scaled Logit Distillation for Ternary Weight Generative Language Modelscited by 20
  17. 2022Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoderscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 36; 2026: 32.

80 papers.

Sort:

By area:

  1. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  2. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  3. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 47
  4. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 32
  5. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  6. 2024Scalable MatMul-free Language Modelingcited by 21
  7. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 20
  8. 2025Bitnet.cpp: Efficient Edge Inference for Ternary LLMscited by 17
  9. 2025FPTQuant: Function-Preserving Transforms for LLM Quantizationcited by 16
  10. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  11. 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inferencecited by 14
  12. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Surveycited by 13
38 more
  1. 2025Matryoshka Quantizationcited by 11
  2. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  3. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  4. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 11
  5. 2025ICQuant: Index Coding enables Low-bit LLM Quantizationcited by 9
  6. 2025A Survey on Efficient Large Language Model Training: From Data-centric Perspectivescited by 9
  7. 2026What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studycited by 9
  8. 2026Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wildcited by 9
  9. 2025Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMscited by 8
  10. 2024OAC: Output-adaptive Calibration for Accurate Post-training Quantizationcited by 7
  11. 2024LPZero: Language Model Zero-cost Proxy Search from Zerocited by 7
  12. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMscited by 7
  13. 2025MACKO: Sparse Matrix-Vector Multiplication for Low Sparsitycited by 6
  14. 2025Progressive Binarization with Semi-Structured Pruning for LLMscited by 5
  15. 2025RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Modelscited by 5
  16. 2025TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAscited by 5
  17. 2025Interpreting the Effects of Quantization on LLMscited by 5
  18. 2025Improving the Straight-Through Estimator with Zeroth-Order Informationcited by 5
  19. 2026Gated Relational Alignment via Confidence-based Distillation for Efficient VLMscited by 5
  20. 2024Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantizationcited by 4
  21. 2025QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decompositioncited by 4
  22. 2025TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledgecited by 4
  23. 2025Rotate, Clip, and Partition: Towards W2A4KV4 Quantization by Integrating Rotation and Learnable Non-uniform Quantizercited by 3
  24. 2025Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flashcited by 3
  25. 2025Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retrainingcited by 3
  26. 2026FreeAct: Freeing Activations for LLM Quantizationcited by 3
  27. 2026Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMscited by 3
  28. 2025Boost Post-Training Quantization via Null Space Optimization for Large Language Modelscited by 2
  29. 2025Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Modelscited by 2
  30. 2025SiLQ: Simple Large Language Model Quantization-Aware Trainingcited by 2
  31. 2025PT2-LLM: Post-Training Ternarization for Large Language Modelscited by 2
  32. 2025BitNet Distillationcited by 2
  33. 2026Towards Understanding Best Practices for Quantization of Vision-Language Modelscited by 2
  34. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidthscited by 2
  35. 2026EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillationcited by 2
  36. 2026I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimizationcited by 2
  37. 2025LCD: Advancing Extreme Low-Bit Clustering for Large Language Models via Knowledge Distillationcited by 1
  38. 2025BiVM: Accurate Binarized Neural Network for Efficient Video Mattingcited by 1

50 most cited of 80 All 80 (JSON)