Model-to-Data Distillation for Graph Neural Networks

Graph neural networks (GNNs) increasingly rely on sophisticated architectures and training procedures to achieve desirable properties such as high predictive performance, fairness, and robustness. However, these properties typically remain tied to the models that learn them, limiting their transferability to simpler models and downstream settings. We introduce model-to-data (M2D) distillation, a new distillation paradigm that transfers properties learned by a complex GNN teacher into graph data, enabling simpler models to recover them through standard training. M2D distillation explicitly trades model complexity for data complexity by jointly learning augmented node features and graph structure that encode the teacher's behavior. The resulting graph serves as a persistent medium for knowledge transfer and can be used with different downstream models. We show that M2D distillation enables simple GNNs to approximate the behavior of substantially more sophisticated teachers, including fairness-aware GNNs, Graph Attention Networks, and Graph Transformers, while maintaining comparable predictive performance and transferring desirable properties of the teacher.