mlsanityData to 7 Oct 2026

Graph Knowledge Distillation to Mixture of Experts

arXiv 2406.11919 ↗First posted 17 Jun 2024cs.LG cs.AI stat.MLcited by 2

On the map: Mixture-of-experts → MoE for time series and graphs

Abstract (arXiv)

In terms of accuracy, Graph Neural Networks (GNNs) are the best architectural choice for the node classification task. Their drawback in real-world deployment is the latency that emerges from the neighbourhood processing operation. One solution to the latency issue is to perform knowledge distillation from a trained GNN to a Multi-Layer Perceptron (MLP), where the MLP processes only the features of the node being classified (and possibly some pre-computed structural information). However, the performance of such MLPs in both transductive and inductive settings remains inconsistent for existing knowledge distillation techniques. We propose to address the performance concerns by using a specially-designed student model instead of an MLP. Our model, named Routing-by-Memory (RbM), is a form of Mixture-of-Experts (MoE), with a design that enforces expert specialization. By encouraging each expert to specialize on a certain region on the hidden representation space, we demonstrate experimentally that it is possible to derive considerably more consistent performance across multiple datasets. Code available at https://github.com/Rufaim/routing-by-memory.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MoE for time series and graphs, one of 12 directions in Mixture-of-experts. Builds on 30 of its 42 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

30 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Spatial Mixture-of-Experts24 Nov

Other references · 29

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  2. 2017Inductive Representation Learning on Large Graphscited by 6,284
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  6. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  7. 2014DeepWalk: Online Learning of Social Representationscited by 1,543
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  9. 2018Pitfalls of Graph Neural Network Evaluationcited by 994
  10. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  11. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  12. 2021Training Graph Neural Networks with 1000 Layerscited by 169
17 more
  1. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  2. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 151
  3. 2021Graph Condensation for Graph Neural Networkscited by 150
  4. 2019Large Memory Layers with Product Keyscited by 120
  5. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  6. 2021Graph-MLP: Node Classification without Message Passing in Graphcited by 73
  7. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Frameworkcited by 69
  8. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
  9. 2021Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoodscited by 57
  10. 2023Quantifying the Knowledge in GNNs for Reliable Distillation into MLPscited by 25
  11. 2023Improving Expert Specialization in Mixture of Expertscited by 16
  12. 2022NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphscited by 14
  13. 2022Sparsely-gated Mixture-of-Expert Layers for CNN Interpretabilitycited by 13
  14. 2022HMOE: Hypernetwork-based Mixture of Experts for Domain Generalizationcited by 8
  15. 2023DRGCN: Dynamic Evolving Initial Residual for Deep Graph Convolutional Networkscited by 6
  16. 2024Classifying Nodes in Graphs without GNNscited by 2
  17. 2023AdaEnsemble: Learning Adaptively Sparse Structured Ensemble Network for Click-Through Rate Predictioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillationcited by 1
  2. 2026PreGS: A Parameter-Transfer-Based Multi-Expert Graph Neural Network for Node Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation
  2. 2022Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classification
  3. 2022SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP
  4. 2025Training Diverse Graph Experts for Ensembles: A Systematic Empirical Study
  5. 2026Learning How Much to Think: Difficulty-Aware Dynamic MoEs for Graph Node Classification
  6. 2025One Router to Route Them All: Homogeneous Expert Routing for Heterogeneous Graph Transformers
  7. 2025GMoPE:A Prompt-Expert Mixture Framework for Graph Foundation Models
  8. 2023Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling