mlsanityData to 14 Oct 2026

HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers

arXiv 2302.09632 ↗First posted 19 Feb 2023cs.CL cs.LGcited by 25

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Knowledge distillation has been shown to be a powerful model compression approach to facilitate the deployment of pre-trained language models in practice. This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. Despite the practical benefits, task-agnostic distillation is challenging. Since the teacher model has a significantly larger capacity and stronger representation power than the student model, it is very difficult for the student to produce predictions that match the teacher's over a massive amount of open-domain training data. Such a large prediction discrepancy often diminishes the benefits of knowledge distillation. To address this challenge, we propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning. Specifically, we initialize the student model from the teacher model, and iteratively prune the student's neurons until the target width is reached. Such an approach maintains a small discrepancy between the teacher's and student's predictions throughout the distillation process, which ensures the effectiveness of knowledge transfer. Extensive experiments demonstrate that HomoDistil achieves significant improvements on existing baselines.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 54 of its 71 references that are in the map. First posted on arXiv 19 Feb 2023.

Builds on

54 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation24 Apr

Other references · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
38 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  2. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  4. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  6. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  8. 2018Neural Network Acceptability Judgmentscited by 958
  9. 2019Are Sixteen Heads Really Better than One?cited by 816
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  11. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  12. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  13. 2019Importance Estimation for Neural Network Pruningcited by 541
  14. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  15. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  16. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  17. 2019On the Efficacy of Knowledge Distillationcited by 381
  18. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  19. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  20. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
  21. 2019Structured Pruning of Large Language Modelscited by 203
  22. 2021Block Pruning For Faster Transformerscited by 177
  23. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  24. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  25. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 113
  26. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 94
  27. 2019Knowledge Distillation from Internal Representationscited by 87
  28. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  29. 2019Knowledge Distillation via Route Constrained Optimizationcited by 75
  30. 2019Global Sparse Momentum SGD for Pruning Very Deep Neural Networkscited by 70
  31. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  32. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  33. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  34. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  35. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  36. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  37. 2020A Gradient Flow Framework For Analyzing Network Pruningcited by 36
  38. 2020FastFormers: Highly Efficient Transformer Models for Natural Language Understandingcited by 28

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 10; 2025: 6; 2026: 3.

25 papers.

Sort:

By area:

  1. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  2. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  3. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 67
  4. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  5. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
  6. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  7. 2025Wanda++: Pruning Large Language Models via Regional Gradientscited by 26
  8. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  9. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 23
  10. 2023Large Multimodal Model Compression via Efficient Pruning and Distillation at AntGroupcited by 8
  11. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  12. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
13 more
  1. 2023Task-agnostic Distillation of Encoder-Decoder Language Modelscited by 4
  2. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  3. 2025Learning Task-Agnostic Representations through Multi-Teacher Distillationcited by 4
  4. 2024VTrans: Accelerating Transformer Compression with Variational Information Bottleneck based Pruningcited by 3
  5. 2025Theoretical Guarantees for Low-Rank Compression of Deep Neural Networkscited by 3
  6. 2023DONUT-hole: DONUT Sparsification by Harnessing Knowledge and Optimizing Learning Efficiencycited by 1
  7. 2024Comb, Prune, Distill: Towards Unified Pruning for Vision Model Compressioncited by 1
  8. 2025IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretrainingcited by 1
  9. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0
  10. 2024TT-MPD: Test Time Model Pruning and Distillationcited by 0
  11. 2026A Study on Hidden Layer Distillation for Large Language Model Pre-Trainingcited by 0
  12. 2026A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactionscited by 0
  13. 2026KDFP: A first-principles approach to knowledge distillation in large language modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models
  2. 2022AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
  3. 2021ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
  4. 2023ERNIE 3.0 Tiny: Frustratingly Simple Method to Improve Task-Agnostic Distillation Generalization
  5. 2021XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
  6. 2024DistiLLM: Towards Streamlined Distillation for Large Language Models
  7. 2021Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processing
  8. 2021Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm