# Graph-based Knowledge Distillation by Multi-head Attention Network

arXiv: https://arxiv.org/abs/1907.02226 · HTML: https://mlsanity.com/paper/1907.02226/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2019.

## Built on by (most cited first, 30 of 31)

- [Learning Affordance Grounding from Exocentric Images](https://mlsanity.com/paper/2203.09905/) (2022)
- [Multimodal learning with graphs](https://mlsanity.com/paper/2209.03299/) (2022)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Distilling Object Detectors with Feature Richness](https://mlsanity.com/paper/2111.00674/) (2021)
- [Graph Neural Networks in IoT: A Survey](https://mlsanity.com/paper/2203.15935/) (2022)
- [Distilling Holistic Knowledge with Graph Neural Networks](https://mlsanity.com/paper/2108.05507/) (2021)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [Black-box Few-shot Knowledge Distillation](https://mlsanity.com/paper/2207.12106/) (2022)
- [NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphs](https://mlsanity.com/paper/2208.10010/) (2022)
- [A Comprehensive Review of Knowledge Distillation in Computer Vision](https://mlsanity.com/paper/2404.00936/) (2024)
- [A Survey on Symbolic Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2408.10210/) (2024)
- [Semi-Supervised Domain Generalizable Person Re-Identification](https://mlsanity.com/paper/2108.05045/) (2021)
- [Graph Flow: Cross-layer Graph Flow Distillation for Dual Efficient Medical Image Segmentation](https://mlsanity.com/paper/2203.08667/) (2022)
- [DistPro: Searching A Fast Knowledge Distillation Process via Meta Optimization](https://mlsanity.com/paper/2204.05547/) (2022)
- [A Survey on AI Sustainability: Emerging Trends on Learning Algorithms and Research Challenges](https://mlsanity.com/paper/2205.03824/) (2022)
- [Compacting Deep Neural Networks for Internet of Things: Methods and Applications](https://mlsanity.com/paper/2103.11083/) (2021)
- [Graph-based Knowledge Distillation: A survey and experimental evaluation](https://mlsanity.com/paper/2302.14643/) (2023)
- [Convolutional Fine-Grained Classification with Self-Supervised Target Relation Regularization](https://mlsanity.com/paper/2208.01997/) (2022)
- [TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation](https://mlsanity.com/paper/2412.11180/) (2024)
- [One-stage Modality Distillation for Incomplete Multimodal Learning](https://mlsanity.com/paper/2309.08204/) (2023)
- [Classifying Nodes in Graphs without GNNs](https://mlsanity.com/paper/2402.05934/) (2024)
- [Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better](https://mlsanity.com/paper/2109.12507/) (2021)
- [Common Knowledge Learning for Generating Transferable Adversarial Examples](https://mlsanity.com/paper/2307.00274/) (2023)
- [Low-Resolution Chest X-ray Classification via Knowledge Distillation and Multi-task Learning](https://mlsanity.com/paper/2405.13370/) (2024)
- [On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2412.11017/) (2024)
- [Graphs as Tools to Improve Deep Learning Methods](https://mlsanity.com/paper/2110.03999/) (2021)
- [Towards Customized Knowledge Distillation for Chip-Level Dense Image Predictions](https://mlsanity.com/paper/2401.13174/) (2024)
- [Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework](https://mlsanity.com/paper/2403.01079/) (2024)
- [PRG: Prompt-Based Distillation Without Annotation via Proxy Relational Graph](https://mlsanity.com/paper/2408.12248/) (2024)

## Abstract (arXiv)

Knowledge distillation (KD) is a technique to derive optimal performance from a small student network (SN) by distilling knowledge of a large teacher network (TN) and transferring the distilled knowledge to the small SN. Since a role of convolutional neural network (CNN) in KD is to embed a dataset so as to perform a given task well, it is very important to acquire knowledge that considers intra-data relations. Conventional KD methods have concentrated on distilling knowledge in data units. To our knowledge, any KD methods for distilling information in dataset units have not yet been proposed. Therefore, this paper proposes a novel method that enables distillation of dataset-based knowledge from the TN using an attention network. The knowledge of the embedding procedure of the TN is distilled to graph by multi-head attention (MHA), and multi-task learning is performed to give relational inductive bias to the SN. The MHA can provide clear information about the source dataset, which can greatly improves the performance of the SN. Experimental results show that the proposed method is 7.05% higher than the SN alone for CIFAR100, which is 2.46% higher than the state-of-the-art.