mlsanityData to 14 Oct 2026

Cross-Architecture Knowledge Distillation

arXiv 2207.05273 ↗First posted 12 Jul 2022cs.CVcited by 41

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Transformer attracts much attention because of its ability to learn global relations and superior performance. In order to achieve higher performance, it is natural to distill complementary knowledge from Transformer to convolutional neural network (CNN). However, most existing knowledge distillation methods only consider homologous-architecture distillation, such as distilling knowledge from CNN to CNN. They may not be suitable when applying to cross-architecture scenarios, such as from Transformer to CNN. To deal with this problem, a novel cross-architecture knowledge distillation method is proposed. Specifically, instead of directly mimicking output/intermediate features of the teacher, partially cross attention projector and group-wise linear projector are introduced to align the student features with the teacher's in two projected feature spaces. And a multi-view robust training scheme is further presented to improve the robustness and stability of the framework. Extensive experiments show that the proposed method outperforms 14 state-of-the-arts on both small-scale and large-scale datasets.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 35 of its 45 references that are in the map. First posted on arXiv 12 Jul 2022.

Builds on

35 of 45 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Knowledge Distillation via the Target-aware Transformer22 May
  2. 2022Spot-adaptive Knowledge Distillation5 May
  3. 2021Lipschitz Continuity Guided Knowledge Distillation29 Aug
  4. 2021Distilling Knowledge via Knowledge Review19 Apr

Other references · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2020End-to-End Object Detection with Transformerscited by 7,414
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2017Mask R-CNNcited by 7,277
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
19 more
  1. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  4. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  8. 2019Relational Knowledge Distillationcited by 889
  9. 2019Contrastive Representation Distillationcited by 748
  10. 2013Do Deep Nets Really Need to be Deep?cited by 474
  11. 2018A Survey on Deep Transfer Learningcited by 419
  12. 2019A Comprehensive Overhaul of Feature Distillationcited by 348
  13. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 301
  14. 2019Structured Knowledge Distillation for Dense Predictioncited by 279
  15. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  16. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  17. 2020CelebA-Spoof: Large-Scale Face Anti-Spoofing Dataset with Rich Annotationscited by 109
  18. 2019Knowledge Distillation from Internal Representationscited by 87
  19. 2021Co-advise: Cross Inductive Bias Distillationcited by 40

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 9; 2025: 15; 2026: 10.

41 papers.

Sort:

By area:

  1. 2023Distilling Efficient Vision Transformers from CNNs for Semantic Segmentationcited by 18
  2. 2023Knowledge Distillation in Vision Transformers: A Critical Reviewcited by 13
  3. 2024ScaleKD: Strong Vision Transformers Could Be Excellent Teacherscited by 11
  4. 2026Distilling Time Series Foundation Models for Efficient Forecastingcited by 11
  5. 2022Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix Factorizationcited by 10
  6. 2024A Comprehensive Review of Knowledge Distillation in Computer Visioncited by 10
  7. 2023Cross Architecture Distillation for Face Recognitioncited by 9
  8. 2025PocketSR: The Super-Resolution Expert in Your Pocket Mobilescited by 8
  9. 2025TransMamba: Fast Universal Architecture Adaption from Transformers to Mambacited by 6
  10. 2023Towards Efficient Task-Driven Model Reprogramming with Foundation Modelscited by 5
  11. 2024Cross-Architecture Auxiliary Feature Space Translation for Efficient Few-Shot Personalized Object Detectioncited by 5
  12. 2025Perspective-Aware Teaching: Adapting Knowledge for Heterogeneous Distillationcited by 5
29 more
  1. 2023RingMo-lite: A Remote Sensing Multi-task Lightweight Network with CNN-Transformer Hybrid Frameworkcited by 4
  2. 2024Fuse Before Transfer: Knowledge Fusion for Heterogeneous Distillationcited by 4
  3. 2025Exploring Few-Shot Defect Segmentation in General Industrial Scenarios with Metric Learning and Vision Foundation Modelscited by 4
  4. 2025Cross-Architecture Distillation Made Simple with Redundancy Suppressioncited by 4
  5. 2025Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridgecited by 4
  6. 2024CATFace: Cross-Attribute-Guided Transformer with Self-Attention Distillation for Low-Quality Face Recognitioncited by 3
  7. 2024HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classificationcited by 3
  8. 2025Hybrid Knowledge Transfer through Attention and Logit Distillation for On-Device Vision Systems in Agricultural IoTcited by 2
  9. 2025MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignmentcited by 2
  10. 2026X-Distill: Cross-Architecture Vision Distillation for Visuomotor Learningcited by 2
  11. 2024Dual Teacher Knowledge Distillation with Domain Alignment for Face Anti-spoofingcited by 1
  12. 2024Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Expertscited by 1
  13. 2024Performance of computer vision algorithms for fine-grained classification using crowdsourced insect imagescited by 1
  14. 2025Efficient Learned Image Compression Through Knowledge Distillationcited by 1
  15. 2025UHKD: A Unified Framework for Heterogeneous Knowledge Distillation via Frequency-Domain Representationscited by 1
  16. 2026KDPhys: An Attention Guided 3D to 2D Knowledge Distillation for Real-time Video-Based Physiological Measurementcited by 1
  17. 2026Beyond Student: An Asymmetric Network for Neural Network Inheritancecited by 1
  18. 2026Collaborative Learning for Semi-Supervised LiDAR Semantic Segmentationcited by 1
  19. 2023Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalizationcited by 0
  20. 2025Multi-Level Decoupled Relational Distillation for Heterogeneous Architecturescited by 0
  21. 2025A Transformer-in-Transformer Network Utilizing Knowledge Distillation for Image Recognitioncited by 0
  22. 2025Clip4Retrofit: Enabling Real-Time Image Labeling on Edge Devices via Cross-Architecture CLIP Distillationcited by 0
  23. 2025Logit-Based Losses Limit the Effectiveness of Feature Knowledge Distillationcited by 0
  24. 2025Scale-invariant and View-relational Representation Learning for Full Surround Monocular Depthcited by 0
  25. 2026AgriKD: Cross-Architecture Knowledge Distillation for Efficient Leaf Disease Classificationcited by 0
  26. 2026SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentationcited by 0
  27. 2026Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledgecited by 0
  28. 2026Cross-Architecture Foundation-Model Distillation for Edge Flood Segmentationcited by 0
  29. 2026CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning to Project for Cross-Task Knowledge Distillation
  2. 2025Revisiting Cross-Architecture Distillation: Adaptive Dual-Teacher Transfer for Lightweight Video Models
  3. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation
  4. 2025Heterogeneous Complementary Distillation
  5. 2025Cross Knowledge Distillation between Artificial and Spiking Neural Networks
  6. 2023Cumulative Spatial Knowledge Distillation for Vision Transformers
  7. 2025A Comprehensive Survey on Knowledge Distillation
  8. 2024Promoting CNNs with Cross-Architecture Knowledge Distillation for Efficient Monocular Depth Estimation