# VRM: Knowledge Distillation via Virtual Relation Matching arXiv: https://arxiv.org/abs/2502.20760 ยท HTML: https://mlsanity.com/paper/2502.20760/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. - Builds on 71 of its 88 references that are in the map. - First posted on arXiv 28 Feb 2025. ## Placed in - Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/) ## Builds on (most cited first, 30 of 71) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016) - [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020) - [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017) - [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019) - [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020) - [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021) - [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014) - [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018) - [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017) - [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017) - [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017) - [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021) - [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016) - [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016) - [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019) - [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019) - [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017) - [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019) - [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019) - [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022) - [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021) - [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019) - [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019) - [Distilling Knowledge via Knowledge Review](https://mlsanity.com/paper/2104.09044/) (2021) - [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019) - [Learning with Pseudo-Ensembles](https://mlsanity.com/paper/1412.4864/) (2014) - [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018) ## Built on by (most cited first, 6 of 6) - [Cross-Architecture Distillation Made Simple with Redundancy Suppression](https://mlsanity.com/paper/2507.21844/) (2025) - [Tempered Self-Similarity Alignment for Physically Plausible Video Generation](https://mlsanity.com/paper/2605.24962/) (2026) - [Feature Representation Transferring to Lightweight Models via Perception Coherence](https://mlsanity.com/paper/2505.06595/) (2025) - [REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiency](https://mlsanity.com/paper/2602.04677/) (2026) - [Flow-Map Distillation on Relation Manifolds for Image Restoration](https://mlsanity.com/paper/2608.05769/) (2026) - [Difference Feature Map Distillation: Transferring Inter-Sample Relational Knowledge Towards Efficient Transformer-Based Tracking](https://mlsanity.com/paper/2610.05707/) (2026) ## Abstract (arXiv) Knowledge distillation (KD) aims to transfer the knowledge of a more capable yet cumbersome teacher model to a lightweight student model. In recent years, relation-based KD methods have fallen behind, as their instance-matching counterparts dominate in performance. In this paper, we revive relational KD by identifying and tackling several key issues in relation-based methods, including their susceptibility to overfitting and spurious responses. Specifically, we transfer novelly constructed affinity graphs that compactly encapsulate a wealth of beneficial inter-sample, inter-class, and inter-view correlations by exploiting virtual views and relations as a new kind of knowledge. As a result, the student has access to richer guidance signals and stronger regularisation throughout the distillation process. To further mitigate the adverse impact of spurious responses, we prune the affinity graphs by dynamically detaching redundant and unreliable edges. Extensive experiments on CIFAR-100, ImageNet, and MS-COCO datasets demonstrate the superior performance of the proposed virtual relation matching (VRM) method, where it consistently sets new state-of-the-art records over a range of models, architectures, tasks, and set-ups. For instance, VRM for the first time hits 74.0% accuracy for ResNet50-to-MobileNetV2 distillation on ImageNet, and improves DeiT-T by 14.44% on CIFAR-100 with a ResNet56 teacher.