mlsanityData to 7 Oct 2026

Less is More: Task-aware Layer-wise Distillation for Language Model Compression

arXiv 2210.01351 ↗First posted 4 Oct 2022cs.CL cs.AI cs.LGcited by 91

Not on the map. Why?

Abstract (arXiv)

Layer-wise distillation is a powerful tool to compress large models (i.e. teacher models) into small ones (i.e., student models). The student distills knowledge from the teacher by mimicking the hidden representations of the teacher at every intermediate layer. However, layer-wise distillation is difficult. Since the student has a smaller model capacity than the teacher, it is often under-fitted. Furthermore, the hidden representations of the teacher contain redundant information that the student does not necessarily need for the target task's learning. To address these challenges, we propose a novel Task-aware layEr-wise Distillation (TED). TED designs task-aware filters to align the hidden representations of the student and the teacher at each layer. The filters select the knowledge that is useful for the target task from the hidden representations. As such, TED reduces the knowledge gap between the two models and helps the student to fit better on the target task. We evaluate TED in two scenarios: continual pre-training and fine-tuning. TED demonstrates significant and consistent improvements over existing distillation methods in both scenarios. Code is available at https://github.com/cliang1453/task-aware-distillation.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 4 Oct 2022.

Builds on

34 of 57 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
22 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  11. 2018Neural Network Acceptability Judgmentscited by 952
  12. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  13. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  14. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  15. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  16. 2018A Simple Method for Commonsense Reasoningcited by 223
  17. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  18. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  19. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  20. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  21. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37
  22. 2021Follow Your Path: a Progressive Method for Knowledge Distillationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 38; 2025: 25; 2026: 20.

91 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2023A Survey on Model Compression for Large Language Modelscited by 263
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  4. 2023Efficient Large Language Models: A Surveycited by 183
  5. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  7. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  8. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  9. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  10. 2024CodecLM: Aligning Language Models with Tailored Synthetic Datacited by 39
  11. 2023Students Parrot Their Teachers: Membership Inference on Model Distillationcited by 31
  12. 2024A Survey of Small Language Modelscited by 31
38 more
  1. 2023Adapting Large Language Models for Content Moderation: Pitfalls in Data Engineering and Supervised Fine-tuningcited by 28
  2. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
  3. 2023AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compressioncited by 26
  4. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Modelscited by 25
  5. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 25
  6. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuningcited by 24
  7. 2024Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancementcited by 21
  8. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 21
  9. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 20
  10. 2024SLMRec: Distilling Large Language Models into Small for Sequential Recommendationcited by 18
  11. 2024BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillationcited by 16
  12. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  13. 2025A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clonecited by 15
  14. 2024Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessmentcited by 13
  15. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  16. 2023Module-wise Adaptive Distillation for Multimodality Foundation Modelscited by 12
  17. 2024Quantifying Knowledge Distillation Using Partial Information Decompositioncited by 12
  18. 2025LLM-Oriented Token-Adaptive Knowledge Distillationcited by 12
  19. 2024PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairscited by 11
  20. 2024Preference-Guided Refactored Tuning for Retrieval Augmented Code Generationcited by 10
  21. 2025The Efficiency vs. Accuracy Trade-off: Optimizing RAG-Enhanced LLM Recommender Systems Using Multi-Head Early Exitcited by 10
  22. 2024Inference Optimization of Foundation Models on AI Acceleratorscited by 9
  23. 2025SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignmentcited by 9
  24. 2025Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Frameworkcited by 8
  25. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thoughtcited by 7
  26. 2026UniSD: Towards a Unified Self-Distillation Framework for Large Language Modelscited by 6
  27. 2024Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillationcited by 5
  28. 2024Optimising TinyML with Quantization and Distillation of Transformer and Mamba Models for Indoor Localisation on Edge Devicescited by 5
  29. 2023RingMo-lite: A Remote Sensing Multi-task Lightweight Network with CNN-Transformer Hybrid Frameworkcited by 4
  30. 2023Towards the Fundamental Limits of Knowledge Transfer over Finite Domainscited by 4
  31. 2024Adversarial Moment-Matching Distillation of Large Language Modelscited by 4
  32. 2024LaDiMo: Layer-wise Distillation Inspired MoEfiercited by 4
  33. 2024Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunablecited by 4
  34. 2024TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillationcited by 4
  35. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  36. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  37. 2025HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMscited by 4
  38. 2024VTrans: Accelerating Transformer Compression with Variational Information Bottleneck based Pruningcited by 3

50 most cited of 91 All 91 (JSON)