mlsanityData to 14 Oct 2026

ReMem: Mutual Information-Aware Fine-tuning of Pretrained Vision Transformers for Effective Knowledge Distillation

arXiv 2506.23041 ↗First posted 29 Jun 2025cs.LG cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Knowledge distillation from pretrained visual representation models offers an effective approach to improve small, task-specific production models. However, the effectiveness of such knowledge transfer drops significantly when distilling from strong models that are pretrained in a large scale. In this paper, we address this challenge for pretrained Vision Transformers (ViTs) by exploring methods to fine-tune them for more effective knowledge transfer. Motivated by the connection between mutual information and distillation effectiveness, we propose to employ mutual information-aware optimization during finetuning. For small or highly-imbalanced downstream datasets where such optimization becomes less effective, we introduce a simple yet effective heuristic of reweighting MLP blocks. This approach is inspired by our observation that top MLP blocks are primarily responsible for mutual information loss. Our method enables small student models to benefit from those pretrained models among the strongest.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 33 of its 51 references that are in the map. First posted on arXiv 29 Jun 2025.

Builds on

33 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Efficient Knowledge Distillation from Model Checkpoints12 Oct
  2. 2022Knowledge Distillation from A Stronger Teacher21 May
  3. 2021Knowledge distillation: A good teacher is patient and consistent9 Jun
  4. 2021Knowledge Distillation as Semiparametric Inference20 Apr
  5. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillation12 Feb

Other references · 28

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2013Describing Textures in the Wildcited by 2,125
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  9. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  10. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  11. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  12. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
16 more
  1. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  2. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  3. 2018MINE: Mutual Information Neural Estimationcited by 683
  4. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  5. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  6. 2018Do Better ImageNet Models Transfer Better?cited by 565
  7. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  8. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  9. 2019On the Efficacy of Knowledge Distillationcited by 381
  10. 2017Snapshot Ensembles: Train 1, get M for freecited by 350
  11. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  12. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  13. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  14. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  15. 2023Can Neural Network Memorization Be Localized?cited by 69
  16. 2018Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Studentscited by 50

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Attention Distillation: self-supervised vision transformer students need more guidance
  2. 2025Distillation Dynamics: Towards Understanding Feature-Based Distillation in Vision Transformers
  3. 2025Scaling Laws for Data-Efficient Visual Transfer Learning
  4. 2025HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models
  5. 2025Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
  6. 2024On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models
  7. 2026LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models
  8. 2023Generic-to-Specific Distillation of Masked Autoencoders