# Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework

arXiv: https://arxiv.org/abs/2305.10758 · HTML: https://mlsanity.com/paper/2305.10758/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 30 of 32)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018)
- [Geom-GCN: Geometric Graph Convolutional Networks](https://mlsanity.com/paper/2002.05287/) (2020)
- [Graph Neural Networks for Social Recommendation](https://mlsanity.com/paper/1902.07243/) (2019)
- [Adaptive Universal Generalized PageRank Graph Neural Network](https://mlsanity.com/paper/2006.07988/) (2020)
- [Beyond Low-frequency Information in Graph Convolutional Networks](https://mlsanity.com/paper/2101.00797/) (2021)
- [Towards Deeper Graph Neural Networks](https://mlsanity.com/paper/2007.09296/) (2020)
- [Label Propagation for Deep Semi-supervised Learning](https://mlsanity.com/paper/1904.04717/) (2019)
- [A Fair Comparison of Graph Neural Networks for Graph Classification](https://mlsanity.com/paper/1912.09893/) (2019)
- [SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation](https://mlsanity.com/paper/2202.03104/) (2022)
- [Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation](https://mlsanity.com/paper/2110.08727/) (2021)
- [Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation](https://mlsanity.com/paper/2105.08919/) (2021)
- [Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework](https://mlsanity.com/paper/2103.02885/) (2021)
- [On Representation Knowledge Distillation for Graph Neural Networks](https://mlsanity.com/paper/2111.04964/) (2021)
- [GraphMixup: Improving Class-Imbalanced Node Classification on Graphs by Self-supervised Context Prediction](https://mlsanity.com/paper/2106.11133/) (2021)
- [Deep geometric knowledge distillation with graphs](https://mlsanity.com/paper/1911.03080/) (2019)
- [Distilling Self-Knowledge From Contrastive Links to Classify Graph Nodes Without Passing Messages](https://mlsanity.com/paper/2106.08541/) (2021)
- [Are Gradients on Graph Structure Reliable in Gray-box Attacks?](https://mlsanity.com/paper/2208.05514/) (2022)
- [Multi-task Self-distillation for Graph-based Semi-Supervised Learning](https://mlsanity.com/paper/2112.01174/) (2021)

## Built on by (most cited first, 24 of 24)

- [Functional-Group-Based Diffusion for Pocket-Specific Molecule Generation and Elaboration](https://mlsanity.com/paper/2306.13769/) (2023)
- [The Heterophilic Graph Learning Handbook: Benchmarks, Models, Theoretical Analysis, Applications and Challenges](https://mlsanity.com/paper/2407.09618/) (2024)
- [Quantifying the Knowledge in GNNs for Reliable Distillation into MLPs](https://mlsanity.com/paper/2306.05628/) (2023)
- [Re-Dock: Towards Flexible and Realistic Molecular Docking with Diffusion Bridge](https://mlsanity.com/paper/2402.11459/) (2024)
- [Protein 3D Graph Structure Learning for Robust Structure-based Protein Property Prediction](https://mlsanity.com/paper/2310.11466/) (2023)
- [Two Trades is not Baffled: Condensing Graph via Crafting Rational Gradient Matching](https://mlsanity.com/paper/2402.04924/) (2024)
- [A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation](https://mlsanity.com/paper/2403.03483/) (2024)
- [Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Tasks](https://mlsanity.com/paper/2403.01400/) (2024)
- [AdaGMLP: AdaBoosting GNN-to-MLP Knowledge Distillation](https://mlsanity.com/paper/2405.14307/) (2024)
- [MuGSI: Distilling GNNs with Multi-Granularity Structural Information for Graph Classification](https://mlsanity.com/paper/2406.19832/) (2024)
- [Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillation](https://mlsanity.com/paper/2407.14768/) (2024)
- [Exploring Feature-based Knowledge Distillation for Recommender System: A Frequency Perspective](https://mlsanity.com/paper/2411.10676/) (2024)
- [Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inference](https://mlsanity.com/paper/2411.14035/) (2024)
- [Propagate & Distill: Towards Effective Graph Learners Using Propagation-Embracing MLPs](https://mlsanity.com/paper/2311.17781/) (2023)
- [Hierarchical Uncertainty-Aware Graph Neural Network](https://mlsanity.com/paper/2504.19820/) (2025)
- [ProGMLP: A Progressive Framework for GNN-to-MLP Knowledge Distillation with Efficient Trade-offs](https://mlsanity.com/paper/2507.19031/) (2025)
- [InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillation](https://mlsanity.com/paper/2601.08033/) (2026)
- [Unveiling the Unseen Potential of Graph Learning through MLPs: Effective Graph Learners Using Propagation-Embracing MLPs](https://mlsanity.com/paper/2311.11759/) (2023)
- [Preserving Node Distinctness in Graph Autoencoders via Similarity Distillation](https://mlsanity.com/paper/2406.17517/) (2024)
- [Learning to Model Graph Structural Information on MLPs via Graph Structure Self-Contrasting](https://mlsanity.com/paper/2409.05573/) (2024)
- [Enhancing Transformer with GNN Structural Knowledge via Distillation: A Novel Approach](https://mlsanity.com/paper/2503.01888/) (2025)
- [Heuristic Methods are Good Teachers to Distill MLPs for Graph Link Prediction](https://mlsanity.com/paper/2504.06193/) (2025)
- [Efficient Graph Knowledge Distillation from GNNs to Kolmogorov--Arnold Networks via Self-Attention Dynamic Sampling](https://mlsanity.com/paper/2509.00560/) (2025)
- [Diffusion-Assisted Distillation for Self-Supervised Graph Representation Learning with MLPs](https://mlsanity.com/paper/2510.04241/) (2025)

## Abstract (arXiv)

Recent years have witnessed the great success of Graph Neural Networks (GNNs) in handling graph-related tasks. However, MLPs remain the primary workhorse for practical industrial applications due to their desirable inference efficiency and scalability. To reduce their gaps, one can directly distill knowledge from a well-designed teacher GNN to a student MLP, which is termed as GNN-to-MLP distillation. However, the process of distillation usually entails a loss of information, and ``which knowledge patterns of GNNs are more likely to be left and distilled into MLPs?" becomes an important question. In this paper, we first factorize the knowledge learned by GNNs into low- and high-frequency components in the spectral domain and then derive their correspondence in the spatial domain. Furthermore, we identified a potential information drowning problem for existing GNN-to-MLP distillation, i.e., the high-frequency knowledge of the pre-trained GNNs may be overwhelmed by the low-frequency knowledge during distillation; we have described in detail what it represents, how it arises, what impact it has, and how to deal with it. In this paper, we propose an efficient Full-Frequency GNN-to-MLP (FF-G2M) distillation framework, which extracts both low-frequency and high-frequency knowledge from GNNs and injects it into MLPs. Extensive experiments show that FF-G2M improves over the vanilla MLPs by 12.6% and outperforms its corresponding teacher GNNs by 2.6% averaged over six graph datasets and three common GNN architectures.