# Deep geometric knowledge distillation with graphs

arXiv: https://arxiv.org/abs/1911.03080 · HTML: https://mlsanity.com/paper/1911.03080/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2019.

## Built on by (most cited first, 21 of 21)

- [Graph Neural Networks in IoT: A Survey](https://mlsanity.com/paper/2203.15935/) (2022)
- [Distilling Holistic Knowledge with Graph Neural Networks](https://mlsanity.com/paper/2108.05507/) (2021)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Quantifying the Knowledge in GNNs for Reliable Distillation into MLPs](https://mlsanity.com/paper/2306.05628/) (2023)
- [Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework](https://mlsanity.com/paper/2305.10758/) (2023)
- [DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval](https://mlsanity.com/paper/2106.13266/) (2021)
- [Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation](https://mlsanity.com/paper/2107.12972/) (2021)
- [The Geometry of Self-supervised Learning Models and its Impact on Transfer Learning](https://mlsanity.com/paper/2209.08622/) (2022)
- [Graph-based Knowledge Distillation: A survey and experimental evaluation](https://mlsanity.com/paper/2302.14643/) (2023)
- [MuGSI: Distilling GNNs with Multi-Granularity Structural Information for Graph Classification](https://mlsanity.com/paper/2406.19832/) (2024)
- [Adaptive Distillation: Aggregating Knowledge from Multiple Paths for Efficient Distillation](https://mlsanity.com/paper/2110.09674/) (2021)
- [Patch-wise Graph Contrastive Learning for Image Translation](https://mlsanity.com/paper/2312.08223/) (2023)
- [Towards Communication-Efficient and Privacy-Preserving Federated Representation Learning](https://mlsanity.com/paper/2109.14611/) (2021)
- [Channel redundancy and overlap in convolutional neural networks with channel-wise NNK graphs](https://mlsanity.com/paper/2110.11400/) (2021)
- [Frameless Graph Knowledge Distillation](https://mlsanity.com/paper/2307.06631/) (2023)
- [Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation](https://mlsanity.com/paper/2312.15112/) (2023)
- [DNN Quantization with Attention](https://mlsanity.com/paper/2103.13322/) (2021)
- [Study of Manifold Geometry using Multiscale Non-Negative Kernel Graphs](https://mlsanity.com/paper/2210.17475/) (2022)
- [ProGMLP: A Progressive Framework for GNN-to-MLP Knowledge Distillation with Efficient Trade-offs](https://mlsanity.com/paper/2507.19031/) (2025)
- [Graphs as Tools to Improve Deep Learning Methods](https://mlsanity.com/paper/2110.03999/) (2021)
- [Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework](https://mlsanity.com/paper/2403.01079/) (2024)

## Abstract (arXiv)

In most cases deep learning architectures are trained disregarding the amount of operations and energy consumption. However, some applications, like embedded systems, can be resource-constrained during inference. A popular approach to reduce the size of a deep learning architecture consists in distilling knowledge from a bigger network (teacher) to a smaller one (student). Directly training the student to mimic the teacher representation can be effective, but it requires that both share the same latent space dimensions. In this work, we focus instead on relative knowledge distillation (RKD), which considers the geometry of the respective latent spaces, allowing for dimension-agnostic transfer of knowledge. Specifically we introduce a graph-based RKD method, in which graphs are used to capture the geometry of latent spaces. Using classical computer vision benchmarks, we demonstrate the ability of the proposed method to efficiently distillate knowledge from the teacher to the student, leading to better accuracy for the same budget as compared to existing RKD alternatives.