mlsanityData to 14 Oct 2026

Knowledge Transfer with Jacobian Matching

arXiv 1803.00443 ↗First posted 1 Mar 2018cs.LG cs.CVcited by 76

Not on the map. Why?

Cited by 37 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Classical distillation methods transfer representations from a "teacher" neural network to a "student" network by matching their output activations. Recent methods also match the Jacobians, or the gradient of output activations with the input. However, this involves making some ad hoc decisions, in particular, the choice of the loss function. In this paper, we first establish an equivalence between Jacobian matching and distillation with input noise, from which we derive appropriate loss functions for Jacobian matching. We then rely on this analysis to apply Jacobian matching to transfer learning by establishing equivalence of a recent transfer learning procedure to distillation. We then show experimentally on standard image datasets that Jacobian-based penalties improve distillation, robustness to noisy inputs, and transfer learning.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 20; 2023: 14; 2024: 11; 2025: 5; 2026: 9.

76 papers.

Sort:

By area:

  1. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  2. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 158
  3. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 134
  4. 2022Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanationscited by 77
  5. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  6. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformerscited by 65
  7. 2022R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learningcited by 59
  8. 2021A Survey on Green Deep Learningcited by 52
  9. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  10. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  11. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  12. 2021Diversity Matters When Learning From Ensemblescited by 27
38 more
  1. 2024VkD: Improving Knowledge Distillation using Orthogonal Projectionscited by 26
  2. 2023Learngene: Inheriting Condensed Knowledge from the Ancestry Model to Descendant Modelscited by 24
  3. 2022Spot-adaptive Knowledge Distillationcited by 22
  4. 2022TaSIL: Taylor Series Imitation Learningcited by 21
  5. 2023Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustnesscited by 20
  6. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 20
  7. 2021Lipschitz Continuity Guided Knowledge Distillationcited by 19
  8. 2021Boosting Active Learning via Improving Test Performancecited by 19
  9. 2023Coaching a Teachable Studentcited by 19
  10. 2021Dead Pixel Test Using Effective Receptive Fieldcited by 18
  11. 2024Good Teachers Explain: Explanation-Enhanced Knowledge Distillationcited by 14
  12. 2021Prune Your Model Before Distill Itcited by 12
  13. 2022Generalized Knowledge Distillation via Relationship Matchingcited by 12
  14. 2021Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillationcited by 11
  15. 2021Joint Matrix Decomposition for Deep Convolutional Neural Networks Compressioncited by 10
  16. 2021Matching Learned Causal Effects of Neural Networks with Domain Priorscited by 10
  17. 2021MixACM: Mixup-Based Robustness Transfer via Distillation of Activated Channel Mapscited by 9
  18. 2024GLiRA: Black-Box Membership Inference Attack via Knowledge Distillationcited by 7
  19. 2022A Closer Look at Knowledge Distillation with Features, Logits, and Gradientscited by 6
  20. 2022Non-Local Latent Relation Distillation for Self-Adaptive 3D Human Pose Estimationcited by 6
  21. 2022Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbationcited by 6
  22. 2024Can Distillation Mitigate Backdoor Attacks in Pre-trained Encoders?cited by 6
  23. 2022Gradient Knowledge Distillation for Pre-trained Language Modelscited by 5
  24. 2023Towards the Fundamental Limits of Knowledge Transfer over Finite Domainscited by 4
  25. 2026Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Modelscited by 4
  26. 2021Contrastive Embeddings for Neural Architecturescited by 3
  27. 2023Characterizing Data Point Vulnerability via Average-Case Robustnesscited by 3
  28. 2023Targeted Activation Penalties Help CNNs Ignore Spurious Signalscited by 3
  29. 2025HDC: Hierarchical Distillation for Multi-level Noisy Consistency in Semi-Supervised Fetal Ultrasound Segmentationcited by 3
  30. 2021Student-Teacher Learning from Clean Inputs to Noisy Inputscited by 2
  31. 2021Similarity Transfer for Knowledge Distillationcited by 2
  32. 2022Auto-Transfer: Learning to Route Transferrable Representationscited by 2
  33. 2022Holistic Approach to Measure Sample-level Adversarial Vulnerability and its Utility in Building Trustworthy Systemscited by 2
  34. 2022Revisiting Architecture-aware Knowledge Distillation: Smaller Models and Faster Searchcited by 2
  35. 2026Distilling Lightweight Domain Experts from Large ML Models by Identifying Relevant Subspacescited by 2
  36. 2021Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognitioncited by 1
  37. 2022Selective Cross-Task Distillationcited by 1
  38. 2022Debias the Black-box: A Fair Ranking Framework via Knowledge Distillationcited by 1

50 most cited of 76 All 76 (JSON)