mlsanityData to 14 Oct 2026

Anti-Shortcut Distillation via Temporal Negative Knowledge Transfer

arXiv 2608.11789 ↗First posted 12 Aug 2026cs.CVcited by 1

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Knowledge distillation (KD) trains a compact student by attracting it towards a converged teacher. It is silent about which directions the teacher itself learned to suppress: repulsive and bias-aware objectives exist, but none exploits the teacher's own trajectory to identify what the student should avoid. We observe that the missing signal is already encoded in the teacher's optimization trajectory: features that an early-stage teacher emphasizes but that a converged teacher attenuates are precisely the shortcut directions worth pushing the student away from. We instantiate this observation as Anti-Shortcut Distillation (ASD), a push--pull KD framework that treats the converged teacher Tfinal as a positive semantic anchor and an early-checkpoint teacher Tearly as a temporal negative reference. ASD couples two losses: a temporal contrastive loss (Ltc) that places the early-teacher feature as a same-sample negative against in-batch and memory-bank final-teacher features in an InfoNCE objective; and a shortcut suppression loss (Lss) that penalizes student projection onto the top eigenvectors of E[DhDhtop], the uncentered second-moment matrix of early-to-final feature displacements. Across 13 teacher--student pairs on CIFAR-100, ImageNet-100, and TinyImageNet, ASD attains the highest clean top-1 accuracy on more than 10 pairs and outperforms standard KD on 12. On CIFAR-100-C corruption robustness, ASD obtains the lowest mean Corruption Error (86.1,mCE) on the most challenging cross-architecture pair (WRN-40-2→ShuffleNet-V2). Mechanistic diagnostics confirm the intended geometry: the ASD student is systematically anti-aligned with the shortcut direction, while its projection onto the robust subspace is substantially larger (0.45 vs. 0.12).

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 32 of its 38 references that are in the map. First posted on arXiv 12 Aug 2026.

Builds on

32 of 38 references.

References · 32

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  7. 2020Supervised Contrastive Learningcited by 2,829
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  12. 2020Knowledge Distillation: A Surveycited by 1,603
20 more
  1. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  2. 2019Natural Adversarial Examplescited by 1,408
  3. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  4. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  5. 2017A Closer Look at Memorization in Deep Networkscited by 942
  6. 2019Relational Knowledge Distillationcited by 889
  7. 2019Contrastive Representation Distillationcited by 748
  8. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  9. 2019On Variational Bounds of Mutual Informationcited by 559
  10. 2018Born Again Neural Networkscited by 546
  11. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  12. 2022Decoupled Knowledge Distillationcited by 388
  13. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  14. 2017Snapshot Ensembles: Train 1, get M for freecited by 350
  15. 2021Distilling Knowledge via Knowledge Reviewcited by 288
  16. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  17. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  18. 2021Teacher's pet: understanding and mitigating biases in distillationcited by 21
  19. 2023Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentationcited by 8
  20. 2023Debiased Distillation by Transplanting the Last Layercited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Yearcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation
  2. 2025UNDO: Understanding Distillation as Optimization
  3. 2023On student-teacher deviations in distillation: does it pay to disobey?
  4. 2026Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance
  5. 2025On the Generalization vs Fidelity Paradox in Knowledge Distillation
  6. 2025Distilled Circuits: A Mechanistic Study of Internal Restructuring in Knowledge Distillation
  7. 2026Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall
  8. 2026When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation