mlsanityData to 7 Oct 2026

Token-Scaled Logit Distillation for Ternary Weight Generative Language Models

arXiv 2308.06744 ↗First posted 13 Aug 2023cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Generative Language Models (GLMs) have shown impressive performance in tasks such as text generation, understanding, and reasoning. However, the large model size poses challenges for practical deployment. To solve this problem, Quantization-Aware Training (QAT) has become increasingly popular. However, current QAT methods for generative models have resulted in a noticeable loss of accuracy. To counteract this issue, we propose a novel knowledge distillation method specifically designed for GLMs. Our method, called token-scaled logit distillation, prevents overfitting and provides superior learning from the teacher model and ground truth. This research marks the first evaluation of ternary weight quantization-aware training of large-scale GLMs with less than 1.0 degradation in perplexity and achieves enhanced accuracy in tasks like common-sense QA and arithmetic reasoning as well as natural language understanding. Our code is available at https://github.com/aiha-lab/TSLD.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 38 of its 50 references that are in the map. First posted on arXiv 13 Aug 2023.

Builds on

38 of 50 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
26 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2020GLU Variants Improve Transformercited by 1,724
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  11. 2018Neural Network Acceptability Judgmentscited by 952
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  13. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  14. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  15. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  16. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  17. 2016Trained Ternary Quantizationcited by 203
  18. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  19. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  20. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  21. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  22. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  23. 2022Extreme Compression for Pre-trained Transformers Made Simple and Efficientcited by 19
  24. 2021KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantizationcited by 14
  25. 2022Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoderscited by 11
  26. 2023Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformerscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 5; 2026: 5.

20 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2023Efficient Large Language Models: A Surveycited by 183
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  4. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  5. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  6. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 60
  7. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 20
  8. 2024Squat: Quant Small Language Models on the Edgecited by 16
  9. 2024Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignmentcited by 9
  10. 2024RILQ: Rank-Insensitive LoRA-based Quantization Error Compensation for Boosting 2-bit Large Language Model Accuracycited by 9
  11. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMscited by 7
  12. 2026Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillationcited by 5
8 more
  1. 2024How to Parameterize Asymmetric Quantization Ranges for Quantization-Aware Trainingcited by 2
  2. 2025BitNet Distillationcited by 2
  3. 2025From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMscited by 1
  4. 2025What Should Feature Distillation Transfer in LLMs? A Task-Tangent Geometry Viewcited by 1
  5. 2025APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Accelerationcited by 1
  6. 2026STAR: Similarity-guided Teacher-Assisted Refinement for Super-Tiny Function Calling Modelscited by 0
  7. 2026Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Modelscited by 0
  8. 2026JustQuant: You Don't Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  2. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
  4. 2024Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  6. 2026QuantLRM: Quantization of Large Reasoning Models via Fine-Tuning Signals
  7. 2025LLM Compression: How Far Can We Go in Balancing Size and Performance?
  8. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data