mlsanityData to 7 Oct 2026

Progressive2: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression

arXiv 2608.00129 ↗First posted 31 Jul 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation (KD) is a widely utilized technique for transferring knowledge from a large model (the teacher) to a smaller model (the student). Owing to its flexibility and broad applicability, KD has been extensively applied in the compression of server-side models to meet the Quality of Service (QoS) requirements of client users. Despite significant advancements, the performance of distillation is substantially compromised when a large disparity exists between the capabilities of the server and the requirements of the client. To alleviate this problem, we propose a novel distillation approach, named Progressive2, which operates through the combination of a progressively stronger teacher and a progressively smaller student. On the side of the teacher, rather than involving all layers simultaneously, we progressively select additional layers for distillation following a raw-to-rich semantic progression, establishing a systematic learning curriculum. Furthermore, we design a teacher-side multi-feature fusion adapter for the teacher to improve training stability, which is theoretically supported by the framework of Lipschitz continuity. On the side of the student, rather than directly training a tiny model, we gradually reduce the size of the network to facilitate an iterative co-evolution with the teacher. Progressive2 serves as a flexible framework; the progressive strategy of the teacher can be deployed independently to achieve an optimal balance between accuracy and training efficiency, while the joint integration of the teacher and the student yields further improvements in overall performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 58 of its 80 references that are in the map. First posted on arXiv 31 Jul 2026.

Builds on

58 of 80 references.

References · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  12. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
38 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  2. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  5. 2019When Does Label Smoothing Help?cited by 985
  6. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  7. 2019Contrastive Representation Distillationcited by 734
  8. 2018Rethinking the Value of Network Pruningcited by 595
  9. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  10. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  11. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  12. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  13. 2022Decoupled Knowledge Distillationcited by 380
  14. 2019On the Efficacy of Knowledge Distillationcited by 377
  15. 2024The Platonic Representation Hypothesiscited by 362
  16. 2018How Does Batch Normalization Help Optimization?cited by 355
  17. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  18. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  19. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  20. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 269
  21. 2018An Empirical Model of Large-Batch Trainingcited by 268
  22. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  23. 2015Unifying distillation and privileged informationcited by 218
  24. 2022Knowledge Distillation from A Stronger Teachercited by 204
  25. 2021Does Knowledge Distillation Really Work?cited by 187
  26. 2019Deep Learning for Spatio-Temporal Data Mining: A Surveycited by 171
  27. 2020Cross-Layer Distillation with Semantic Calibrationcited by 141
  28. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  29. 2022Curriculum Temperature for Knowledge Distillationcited by 110
  30. 2022Masked Generative Distillationcited by 107
  31. 2015Reducing Overfitting in Deep Networks by Decorrelating Representationscited by 94
  32. 2019Deep Learning vs. Traditional Computer Visioncited by 94
  33. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  34. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  35. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 82
  36. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 76
  37. 2023FAMO: Fast Adaptive Multitask Optimizationcited by 74
  38. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73

50 most cited of 58 All 58 (JSON)

Built on by

Not cited yet.