mlsanityData to 14 Oct 2026

Large Language Model Meets Graph Neural Network in Knowledge Distillation

arXiv 2402.05894 ↗First posted 8 Feb 2024cs.AI cs.LGcited by 14

On the map: Knowledge distillation → Graph neural network distillation

Abstract (arXiv)

In service-oriented architectures, accurately predicting the Quality of Service (QoS) is crucial for maintaining reliability and enhancing user satisfaction. However, significant challenges remain due to existing methods always overlooking high-order latent collaborative relationships between users and services and failing to dynamically adjust feature learning for every specific user-service invocation, which are critical for learning accurate features. Additionally, reliance on RNNs for capturing QoS evolution hampers models' ability to detect long-term trends due to difficulties in managing long-range dependencies. To address these challenges, we propose the underlineTarget-Prompt underlineOnline underlineGraph underlineCollaborative underlineLearning (TOGCL) framework for temporal-aware QoS prediction. TOGCL leverages a dynamic user-service invocation graph to model historical interactions, providing a comprehensive representation of user-service relationships. Building on this graph, it develops a target-prompt graph attention network to extract online deep latent features of users and services at each time slice, simultaneously considering implicit collaborative relationships between target users/services and their neighbors, as well as relevant historical QoS values. Additionally, a multi-layer Transformer encoder is employed to uncover temporal feature evolution patterns of users and services, leading to temporal-aware QoS prediction. Extensive experiments conducted on the WS-DREAM dataset demonstrate that our proposed TOGCL framework significantly outperforms state-of-the-art methods across multiple metrics, achieving improvements of up to 38.80%. These results underscore the effectiveness of the TOGCL framework for precise temporal QoS prediction.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation. Builds on 60 of its 85 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

60 of 85 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023STRIDE: Structure and Embedding Distillation with Attention for Graph Neural Networks24 Oct
  2. 2023VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs4 Aug
  3. 2023Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs24 Mar
  4. 2022SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP18 Oct
  5. 2022Compressing Deep Graph Neural Networks via Adversarial Knowledge Distillation24 May
  6. 2021On Representation Knowledge Distillation for Graph Neural Networks9 Nov
  7. 2021Graph-Free Knowledge Distillation for Graph Neural Networks16 May

Other references · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2017Graph Attention Networkscited by 8,923
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2017Inductive Representation Learning on Large Graphscited by 6,320
  9. 2018How Powerful are Graph Neural Networks?cited by 4,393
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2020Longformer: The Long-Document Transformercited by 3,413
  12. 2023Mistral 7Bcited by 3,164
38 more
  1. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  3. 2019Simplifying Graph Convolutional Networkscited by 1,477
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 976
  6. 2020Simple and Deep Graph Convolutional Networkscited by 860
  7. 2019Contrastive Representation Distillationcited by 748
  8. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  9. 2020A Generalization of Transformer Networks to Graphscited by 661
  10. 2020Contrastive Learning with Hard Negative Samplescited by 628
  11. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  12. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  13. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 506
  14. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  15. 2021Beyond Low-frequency Information in Graph Convolutional Networkscited by 362
  16. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 274
  17. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 252
  18. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 250
  19. 2023Language is All a Graph Needscited by 220
  20. 2021BernNet: Learning Arbitrary Graph Spectral Filters via Bernstein Approximationcited by 195
  21. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 176
  22. 2023LLMRec: Large Language Models with Graph Augmentation for Recommendationcited by 172
  23. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  24. 2022Revisiting Heterophily For Graph Neural Networkscited by 169
  25. 2021Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Predictioncited by 138
  26. 2023SGFormer: Simplifying and Empowering Transformers for Large-Graph Representationscited by 133
  27. 2020Distilling Knowledge from Graph Convolutional Networkscited by 129
  28. 2023A Survey of Graph Meets Large Language Model: Progress and Future Directionscited by 85
  29. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  30. 2023SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learningcited by 73
  31. 2021Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classificationcited by 67
  32. 2023Can Large Language Models Empower Molecular Property Prediction?cited by 44
  33. 2020Model Extraction Attacks on Graph Neural Networks: Taxonomy and Realizationcited by 40
  34. 2023Prompt-based Node Feature Extractor for Few-shot Learning on Text-Attributed Graphscited by 23
  35. 2021Higher-Order Attribute-Enhancing Heterogeneous Graph Neural Networkscited by 22
  36. 2023An Empirical Study of Instruction-tuning Large Language Models in Chinesecited by 17
  37. 2024An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaborationcited by 16
  38. 2023TouchUp-G: Improving Feature Representation through Graph-Centric Finetuningcited by 12

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 6.

14 papers.

Sort:

By area:

  1. 2024LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generationcited by 15
  2. 2024A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspectivecited by 9
  3. 2025Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigationcited by 5
  4. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  5. 2025Large Language Models for Knowledge Graph Embedding: A Surveycited by 3
  6. 2025From Text to Graph: Leveraging Graph Neural Networks for Enhanced Explainability in NLPcited by 1
  7. 2025Knowledge Graph-Guided Multi-Agent Distillation for Reliable Industrial Question Answering with Datasetscited by 1
  8. 2025Preference-driven Knowledge Distillation for Few-shot Node Classificationcited by 1
  9. 2026PRECTR-V2:Unified Relevance-CTR Framework with Cross-User Preference Mining, Exposure Bias Correction, and LLM-Distilled Encoder Optimizationcited by 1
  10. 2026Detecting Miscitation on the Scholarly Web through LLM-Augmented Text-Rich Graph Learningcited by 0
  11. 2026Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Modelscited by 0
  12. 2026Patients-like-me: A Variational LM--GNN Framework for Explainable Clinical Predictioncited by 0
2 more
  1. 2026Data-Efficient Agentic Graph Domain Adaptation via Reliability-Aware Prototype Learningcited by 0
  2. 2026DeKoR: Decomposed Knowledge Routing for Efficient GNN--LLM Fusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025QoSDiff: An Implicit Topological Embedding Learning Framework Leveraging Denoising Diffusion and Adversarial Attention for Robust QoS Prediction
  2. 2026Self-Augmented Mixture-of-Experts for QoS Prediction
  3. 2025Hephaestus: Mixture Generative Modeling with Energy Guidance for Large-scale QoS Degradation
  4. 2025RLHGNN: Reinforcement Learning-driven Heterogeneous Graph Neural Network for Next Activity Prediction in Business Processes
  5. 2026TemporalGraphLLM: Temporal Graph Neural Networks with Large Language Models for Dynamic Text-Attributed Graphs
  6. 2025Identifying Appropriately-Sized Services with Deep Reinforcement Learning
  7. 2025GLOW: Graph-Language Co-Encoding for Agentic Workflow Performance Prediction
  8. 2024Retrieval-Augmented Generation Meets Data-Driven Tabula Rasa Approach for Temporal Knowledge Graph Forecasting