mlsanityData to 14 Oct 2026

Knowledge distillation

12 directions · 3,513 papers · 34 this week

Busiest this week

Most new since Mon 12 Oct

On-policy and self-distillation for LLMs

446 papers22 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

  1. Do As I Do, Not As I Say: On-Policy Distillation Inherits the Teacher in Vision-Language Models
  2. Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
  3. VASPO: Self-Distillation with Privileged Value Estimation
  4. RM-OPD: On-Policy Distillation for Generative Reward Models
  5. ActionOPD: Distilling Decisions, Not Tokens
Open →

All directions

Largest first · week from Mon 12 Oct

  1. General knowledge distillation methodsknowledge, distillation, teacher, studentLow confidence in this grouping: the papers may belong to more than one topic, or the group may change · About9003Low confidence in this grouping: the papers may belong to more than one topic, or the group may change
  2. Knowledge distillation for LLMslanguage, distillation, knowledge, reasoningHigh confidence in this grouping: the papers form one clear topic · About6236High confidence in this grouping: the papers form one clear topic
  3. Detection and dense prediction distillationdistillation, detection, knowledge, objectMedium confidence in this grouping: mostly one topic, with blurry edges · About4510Medium confidence in this grouping: mostly one topic, with blurry edges
  4. On-policy and self-distillation for LLMson-policy, opd, teacher, on-policy distillationHigh confidence in this grouping: the papers form one clear topic · About44622High confidence in this grouping: the papers form one clear topic
  5. Multimodal distillationdistillation, visual, multimodal, knowledgeLow confidence in this grouping: the papers may belong to more than one topic, or the group may change · About3082Low confidence in this grouping: the papers may belong to more than one topic, or the group may change
  6. Speech and audio distillationspeech, distillation, audio, knowledgeHigh confidence in this grouping: the papers form one clear topic · About1971High confidence in this grouping: the papers form one clear topic
  7. Federated, robust and private distillationfederated, knowledge, adversarial, federated learningMedium confidence in this grouping: mostly one topic, with blurry edges · About1910Medium confidence in this grouping: mostly one topic, with blurry edges
  8. Distillation for medical imagingsegmentation, medical, knowledge, distillationMedium confidence in this grouping: mostly one topic, with blurry edges · About1910Medium confidence in this grouping: mostly one topic, with blurry edges
  9. Graph neural network distillationgraph, gnns, knowledge, gnnHigh confidence in this grouping: the papers form one clear topic · About980High confidence in this grouping: the papers form one clear topic
  10. Distillation for continual learningcontinual, forgetting, continual learning, catastrophic forgettingHigh confidence in this grouping: the papers form one clear topic · About510High confidence in this grouping: the papers form one clear topic
  11. Distillation for recommender systemsrecommendation, knowledge, user, recommenderHigh confidence in this grouping: the papers form one clear topic · About340High confidence in this grouping: the papers form one clear topic
  12. Distillation for spiking neural networkssnns, spiking, snn, neuralHigh confidence in this grouping: the papers form one clear topic · About230High confidence in this grouping: the papers form one clear topic

Weekly activity

Papers per week, 29 Jun – 12 Oct (last week partial)

  1. DirectionConfidencepapers16 wks
  2. On-policy and self-distillation for LLMsHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 29 Jun: 16; 6 Jul: 10; 13 Jul: 11; 20 Jul: 8; 27 Jul: 18; 3 Aug: 32; 10 Aug: 23; 17 Aug: 9; 24 Aug: 20; 31 Aug: 17; 7 Sep: 14; 14 Sep: 12; 21 Sep: 36; 28 Sep: 96; 5 Oct: 42; 12 Oct: 22446386
  3. Multimodal distillationLow confidence in this grouping: the papers may belong to more than one topic, or the group may change · AboutLow confidence in this grouping: the papers may belong to more than one topic, or the group may changePer week: 29 Jun: 4; 6 Jul: 4; 13 Jul: 2; 20 Jul: 2; 27 Jul: 7; 3 Aug: 4; 10 Aug: 4; 17 Aug: 7; 24 Aug: 3; 31 Aug: 3; 7 Sep: 3; 14 Sep: 3; 21 Sep: 6; 28 Sep: 25; 5 Oct: 5; 12 Oct: 230884
  4. Knowledge distillation for LLMsHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 29 Jun: 3; 6 Jul: 3; 13 Jul: 1; 20 Jul: 3; 27 Jul: 3; 3 Aug: 2; 10 Aug: 2; 17 Aug: 2; 24 Aug: 6; 31 Aug: 3; 7 Sep: 2; 14 Sep: 6; 21 Sep: 7; 28 Sep: 4; 5 Oct: 13; 12 Oct: 662366
  5. General knowledge distillation methodsLow confidence in this grouping: the papers may belong to more than one topic, or the group may change · AboutLow confidence in this grouping: the papers may belong to more than one topic, or the group may changePer week: 29 Jun: 4; 6 Jul: 3; 13 Jul: 5; 20 Jul: 1; 27 Jul: 2; 3 Aug: 3; 10 Aug: 4; 17 Aug: 1; 24 Aug: 8; 31 Aug: 2; 7 Sep: 2; 14 Sep: 2; 21 Sep: 2; 28 Sep: 4; 5 Oct: 5; 12 Oct: 390051
  6. Detection and dense prediction distillationMedium confidence in this grouping: mostly one topic, with blurry edges · AboutMedium confidence in this grouping: mostly one topic, with blurry edgesPer week: 29 Jun: 4; 6 Jul: 1; 13 Jul: 2; 20 Jul: 2; 27 Jul: 3; 3 Aug: 1; 10 Aug: 2; 17 Aug: 2; 24 Aug: 2; 31 Aug: 3; 7 Sep: 2; 14 Sep: 2; 21 Sep: 3; 28 Sep: 4; 5 Oct: 2; 12 Oct: 045135
  7. Speech and audio distillationHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 29 Jun: 0; 6 Jul: 0; 13 Jul: 1; 20 Jul: 1; 27 Jul: 0; 3 Aug: 0; 10 Aug: 2; 17 Aug: 1; 24 Aug: 1; 31 Aug: 1; 7 Sep: 1; 14 Sep: 3; 21 Sep: 1; 28 Sep: 4; 5 Oct: 0; 12 Oct: 119717
  8. Distillation for continual learningHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 29 Jun: 0; 6 Jul: 0; 13 Jul: 0; 20 Jul: 0; 27 Jul: 0; 3 Aug: 1; 10 Aug: 0; 17 Aug: 0; 24 Aug: 0; 31 Aug: 1; 7 Sep: 0; 14 Sep: 0; 21 Sep: 0; 28 Sep: 3; 5 Oct: 1; 12 Oct: 0516
  9. Distillation for medical imagingMedium confidence in this grouping: mostly one topic, with blurry edges · AboutMedium confidence in this grouping: mostly one topic, with blurry edgesPer week: 29 Jun: 0; 6 Jul: 0; 13 Jul: 2; 20 Jul: 0; 27 Jul: 3; 3 Aug: 0; 10 Aug: 1; 17 Aug: 1; 24 Aug: 0; 31 Aug: 2; 7 Sep: 2; 14 Sep: 1; 21 Sep: 0; 28 Sep: 1; 5 Oct: 2; 12 Oct: 019115
  10. Federated, robust and private distillationMedium confidence in this grouping: mostly one topic, with blurry edges · AboutMedium confidence in this grouping: mostly one topic, with blurry edgesPer week: 29 Jun: 0; 6 Jul: 0; 13 Jul: 1; 20 Jul: 0; 27 Jul: 1; 3 Aug: 1; 10 Aug: 1; 17 Aug: 0; 24 Aug: 1; 31 Aug: 1; 7 Sep: 1; 14 Sep: 1; 21 Sep: 2; 28 Sep: 0; 5 Oct: 0; 12 Oct: 019110
  11. Graph neural network distillationHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 29 Jun: 1; 6 Jul: 0; 13 Jul: 0; 20 Jul: 0; 27 Jul: 0; 3 Aug: 0; 10 Aug: 0; 17 Aug: 0; 24 Aug: 0; 31 Aug: 0; 7 Sep: 0; 14 Sep: 0; 21 Sep: 0; 28 Sep: 0; 5 Oct: 1; 12 Oct: 0982
  12. Distillation for recommender systemsHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 29 Jun: 2; 6 Jul: 0; 13 Jul: 0; 20 Jul: 0; 27 Jul: 0; 3 Aug: 0; 10 Aug: 0; 17 Aug: 2; 24 Aug: 0; 31 Aug: 1; 7 Sep: 0; 14 Sep: 0; 21 Sep: 0; 28 Sep: 0; 5 Oct: 0; 12 Oct: 0345
  13. Distillation for spiking neural networksHigh confidence in this grouping: the papers form one clear topic · AboutHigh confidence in this grouping: the papers form one clear topicPer week: 29 Jun: 0; 6 Jul: 0; 13 Jul: 0; 20 Jul: 0; 27 Jul: 0; 3 Aug: 0; 10 Aug: 0; 17 Aug: 0; 24 Aug: 1; 31 Aug: 0; 7 Sep: 0; 14 Sep: 0; 21 Sep: 0; 28 Sep: 0; 5 Oct: 0; 12 Oct: 0231