mlsanityData to 14 Oct 2026

BitNet Distillation

arXiv 2510.13998 ↗First posted 15 Oct 2025cs.LG cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

In this paper, we present BitNet Distillation (BitDistill), a lightweight pipeline that fine-tunes off-the-shelf full-precision LLMs (e.g., Qwen) into 1.58-bit precision (i.e., ternary weights {-1, 0, 1}) for specific downstream tasks, achieving strong task-specific performance with minimal computational cost. Specifically, BitDistill incorporates three key techniques: the SubLN module, as introduced in BitNet; multi-head attention distillation, based on MiniLM; and continual pre-training, which serves as a crucial warm-up step to mitigate the scalability issue of the performance gap between finetuned full-precision and 1.58-bit LLMs on specific tasks. Experimental results show that BitDistill achieves performance comparable to the full-precision counterpart models across model size, while enabling up to 10x memory savings and 2.65x faster inference on CPUs. Code is available at https://github.com/microsoft/BitNet.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

31 of 39 references.

References · 31

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
19 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  2. 2015Teaching Machines to Read and Comprehendcited by 1,049
  3. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  4. 2023A Survey on Large Language Models for Recommendationcited by 379
  5. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  6. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
  7. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  8. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  9. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  10. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  11. 2023Representation Learning with Large Language Models for Recommendationcited by 155
  12. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  13. 2023Text Classification via Large Language Modelscited by 106
  14. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  15. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 52
  16. 2025BitNet b1.58 2B4T Technical Reportcited by 38
  17. 2023Token-Scaled Logit Distillation for Ternary Weight Generative Language Modelscited by 20
  18. 2025Large Language Models For Text Classification: Case Study And Comprehensive Reviewcited by 12
  19. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantizationcited by 5
  2. 2026BitNet Text Embeddingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20241-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs
  2. 2025Bitnet.cpp: Efficient Edge Inference for Ternary LLMs
  3. 2024FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation
  4. 2024BitNet a4.8: 4-bit Activations for 1-bit LLMs
  5. 2026Breaking the 1.58-bit Barrier for Ternary LLMs
  6. 2026CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
  7. 2026VitaLLM: A Versatile, Ultra-Compact Ternary LLM Accelerator with Dependency-Aware Scheduling ↗
  8. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs