mlsanityData to 14 Oct 2026

Transferring Inductive Biases through Knowledge Distillation

arXiv 2006.00555 ↗First posted 31 May 2020cs.LG cs.AI stat.MLcited by 41

Not on the map. Why?

Abstract (arXiv)

Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly representative of the conditions at test time. However, defining, designing and efficiently adapting inductive biases is not necessarily straightforward. In this paper, we explore the power of knowledge distillation for transferring the effect of inductive biases from one model to another. We consider families of models with different inductive biases, LSTMs vs. Transformers and CNNs vs. MLPs, in the context of tasks and scenarios where having the right inductive biases is critical. We study the effect of inductive biases on the solutions the models converge to and investigate how and to what extent the effect of inductive biases is transferred through knowledge distillation, in terms of not only performance but also different aspects of converged solutions.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 7; 2023: 13; 2024: 5; 2025: 6; 2026: 4.

41 papers.

Sort:

By area:

  1. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  2. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  3. 2021The Benchmark Lotterycited by 136
  4. 2021A Survey of Visual Transformerscited by 117
  5. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  6. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  7. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 75
  8. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformerscited by 65
  9. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 47
  10. 2023Modeling rapid language learning by distilling Bayesian priors into artificial neural networkscited by 42
  11. 2021Efficient Transformers in Reinforcement Learning using Actor-Learner Distillationcited by 35
  12. 2023Adaptive Computation with Elastic Input Sequencecited by 22
29 more
  1. 2021Forward Compatible Training for Large-Scale Embedding Retrieval Systemscited by 19
  2. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  3. 2023Distillation from Heterogeneous Models for Top-K Recommendationcited by 13
  4. 2023RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixercited by 13
  5. 2022CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classificationcited by 12
  6. 2021Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillationcited by 11
  7. 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
  8. 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoningcited by 9
  9. 2022Co-training 2L Submodels for Visual Recognitioncited by 8
  10. 2025Can Interpretation Predict Behavior on Unseen Data?cited by 8
  11. 2023Adaptivity and Modularity for Efficient Generalization Over Task Complexitycited by 7
  12. 2025Perspective-Aware Teaching: Adapting Knowledge for Heterogeneous Distillationcited by 5
  13. 2025Teasing Apart Architecture and Initial Weights as Sources of Inductive Bias in Neural Networkscited by 5
  14. 2024An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-Trainingcited by 4
  15. 2024A Method on Searching Better Activation Functionscited by 4
  16. 2025Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridgecited by 4
  17. 2026Volume Transformer: Revisiting Vanilla Transformers for 3D Scene Understandingcited by 4
  18. 2023ChapGTP, ILLC's Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formationcited by 3
  19. 2023Efficient and Robust Jet Tagging at the LHC with Knowledge Distillationcited by 3
  20. 2026Effective Distillation to Hybrid xLSTM Architecturescited by 3
  21. 2022Do we need Label Regularization to Fine-tune Pre-trained Language Models?cited by 2
  22. 2022Traffic-Twitter Transformer: A Nature Language Processing-joined Framework For Network-wide Traffic Forecastingcited by 2
  23. 2023Knowledge Distillation for Anomaly Detectioncited by 2
  24. 2024Knowledge Distillation Approaches for Accurate and Efficient Recommender Systemcited by 2
  25. 2025Propositional Logic for Probing Generalization in Neural Networkscited by 1
  26. 2022Adaptive Attention Link-based Regularization for Vision Transformerscited by 0
  27. 2023Bridging Modality Gap for Visual Grounding with Effecitve Cross-modal Distillationcited by 0
  28. 2023On the Necessity of Metalearning: Learning Suitable Parameterizations for Learning Processescited by 0
  29. 2026Accelerating Vision Transformers on Brain Processing Unitcited by 0