mlsanityData to 14 Oct 2026

Preference-driven Knowledge Distillation for Few-shot Node Classification

arXiv 2510.10116 ↗First posted 11 Oct 2025cs.LG cs.SIcited by 1

On the map: Knowledge distillation → Graph neural network distillation

Abstract (arXiv)

Graph neural networks (GNNs) can efficiently process text-attributed graphs (TAGs) due to their message-passing mechanisms, but their training heavily relies on the human-annotated labels. Moreover, the complex and diverse local topologies of nodes of real-world TAGs make it challenging for a single mechanism to handle. Large language models (LLMs) perform well in zero-/few-shot learning on TAGs but suffer from a scalability challenge. Therefore, we propose a preference-driven knowledge distillation (PKD) framework to synergize the complementary strengths of LLMs and various GNNs for few-shot node classification. Specifically, we develop a GNN-preference-driven node selector that effectively promotes prediction distillation from LLMs to teacher GNNs. To further tackle nodes' intricate local topologies, we develop a node-preference-driven GNN selector that identifies the most suitable teacher GNN for each node, thereby facilitating tailored knowledge distillation from teacher GNNs to the student GNN. Extensive experiments validate the efficacy of our proposed framework in few-shot node classification on real-world TAGs. Our code is be available.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation. Builds on 49 of its 65 references that are in the map. First posted on arXiv 11 Oct 2025.

Builds on

49 of 65 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Distilling Large Language Models for Text-Attributed Graph Learning19 Feb
  2. 2024Large Language Model Meets Graph Neural Network in Knowledge Distillation8 Feb
  3. 2023The Devil is in the Data: Learning Fair Graph Neural Networks via Partial Knowledge Distillation29 Nov
  4. 2022Boosting Graph Neural Networks via Adaptive Knowledge Distillation12 Oct

Other references · 45

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2017Graph Attention Networkscited by 8,923
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  9. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  10. 2024Mixtral of Expertscited by 1,823
  11. 2014DeepWalk: Online Learning of Social Representationscited by 1,547
  12. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
33 more
  1. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 993
  2. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 976
  3. 2021E(n) Equivariant Graph Neural Networkscited by 965
  4. 2020Simple and Deep Graph Convolutional Networkscited by 860
  5. 2018Hierarchical Graph Representation Learning with Differentiable Poolingcited by 787
  6. 2020Adaptive Universal Generalized PageRank Graph Neural Networkcited by 595
  7. 2019Graph Transformer Networkscited by 523
  8. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  9. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 432
  10. 2023A critical look at the evaluation of GNNs under heterophily: Are we really making progress?cited by 343
  11. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 274
  12. 2020Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networkscited by 260
  13. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 252
  14. 2023One for All: Towards Training One Graph Model for All Classification Taskscited by 243
  15. 2021GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graphcited by 143
  16. 2023Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learningcited by 133
  17. 2023Edge Directionality Improves Learning on Heterophilic Graphscited by 90
  18. 2023Label-free Node Classification on Graphs with Large Language Models (LLMS)cited by 72
  19. 2019Graph Few-shot Learning via Knowledge Transfercited by 63
  20. 2024LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddingscited by 61
  21. 2021TextGNN: Improving Text Encoder via Graph Neural Network in Sponsored Searchcited by 54
  22. 2023HoloNets: Spectral Convolutions do extend to Directed Graphscited by 20
  23. 2024GAugLLM: Improving Graph Contrastive Learning for Text-Attributed Graphs with Large Language Modelscited by 17
  24. 2022Toward Robust Graph Semi-Supervised Learning against Extreme Data Scarcitycited by 11
  25. 2024Exploring the Potential of Large Language Models for Heterophilic Graphscited by 10
  26. 2024TAGLAS: An atlas of text-attributed graph datasets in the era of large graph and language modelscited by 9
  27. 2023Collaborative Learning via Prediction Consensuscited by 8
  28. 2024Sign is Not a Remedy: Multiset-to-Multiset Message Passing for Learning on Heterophilic Graphscited by 8
  29. 2025HeTGB: A Comprehensive Benchmark for Heterophilic Text-Attributed Graphscited by 5
  30. 2025Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigationcited by 5
  31. 2022Incorporating Heterophily into Graph Neural Networks for Graph Classificationcited by 4
  32. 2025IceBerg: Debiased Self-Training for Class-Imbalanced Node Classificationcited by 4
  33. 2024Path-LLM: A Shortest-Path-based LLM Learning for Unified Graph Representationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Beyond One-Size-Fits-All: Adaptive Subgraph Denoising for Zero-Shot Graph Learning with Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026HopRank: Self-Supervised LLM Preference-Tuning on Graphs for Few-Shot Node Classification
  2. 2023Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs
  3. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework
  4. 2024Enhancing Graph Neural Networks with Limited Labeled Data by Actively Distilling Knowledge from Large Language Models
  5. 2023Graph-based Knowledge Distillation: A survey and experimental evaluation
  6. 2026Beyond the Golden Teacher: Enhancing Graph Learning through LLM-GNN Co-teaching
  7. 2026InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillation
  8. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation