# Large Language Model Meets Graph Neural Network in Knowledge Distillation

arXiv: https://arxiv.org/abs/2402.05894 · HTML: https://mlsanity.com/paper/2402.05894/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 60 of its 85 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 30 of 60)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Revisiting Semi-Supervised Learning with Graph Embeddings](https://mlsanity.com/paper/1603.08861/) (2016)
- [Simple and Deep Graph Convolutional Networks](https://mlsanity.com/paper/2007.02133/) (2020)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Beyond Low-frequency Information in Graph Convolutional Networks](https://mlsanity.com/paper/2101.00797/) (2021)
- [Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs](https://mlsanity.com/paper/2307.03393/) (2023)
- [Can Language Models Solve Graph Problems in Natural Language?](https://mlsanity.com/paper/2305.10037/) (2023)
- [DeeperGCN: All You Need to Train Deeper GCNs](https://mlsanity.com/paper/2006.07739/) (2020)

## Built on by (most cited first, 14 of 14)

- [LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation](https://mlsanity.com/paper/2404.00998/) (2024)
- [A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective](https://mlsanity.com/paper/2403.16137/) (2024)
- [Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigation](https://mlsanity.com/paper/2510.08962/) (2025)
- [Deploying Foundation Model Powered Agent Services: A Survey](https://mlsanity.com/paper/2412.13437/) (2024)
- [Large Language Models for Knowledge Graph Embedding: A Survey](https://mlsanity.com/paper/2501.07766/) (2025)
- [From Text to Graph: Leveraging Graph Neural Networks for Enhanced Explainability in NLP](https://mlsanity.com/paper/2504.02064/) (2025)
- [Knowledge Graph-Guided Multi-Agent Distillation for Reliable Industrial Question Answering with Datasets](https://mlsanity.com/paper/2510.06240/) (2025)
- [Preference-driven Knowledge Distillation for Few-shot Node Classification](https://mlsanity.com/paper/2510.10116/) (2025)
- [PRECTR-V2:Unified Relevance-CTR Framework with Cross-User Preference Mining, Exposure Bias Correction, and LLM-Distilled Encoder Optimization](https://mlsanity.com/paper/2602.20676/) (2026)
- [Detecting Miscitation on the Scholarly Web through LLM-Augmented Text-Rich Graph Learning](https://mlsanity.com/paper/2603.12290/) (2026)
- [Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models](https://mlsanity.com/paper/2605.07783/) (2026)
- [Patients-like-me: A Variational LM--GNN Framework for Explainable Clinical Prediction](https://mlsanity.com/paper/2608.04193/) (2026)
- [Data-Efficient Agentic Graph Domain Adaptation via Reliability-Aware Prototype Learning](https://mlsanity.com/paper/2609.14045/) (2026)
- [DeKoR: Decomposed Knowledge Routing for Efficient GNN--LLM Fusion](https://mlsanity.com/paper/2610.14710/) (2026)

## Abstract (arXiv)

In service-oriented architectures, accurately predicting the Quality of Service (QoS) is crucial for maintaining reliability and enhancing user satisfaction. However, significant challenges remain due to existing methods always overlooking high-order latent collaborative relationships between users and services and failing to dynamically adjust feature learning for every specific user-service invocation, which are critical for learning accurate features. Additionally, reliance on RNNs for capturing QoS evolution hampers models' ability to detect long-term trends due to difficulties in managing long-range dependencies. To address these challenges, we propose the underlineTarget-Prompt underlineOnline underlineGraph underlineCollaborative underlineLearning (TOGCL) framework for temporal-aware QoS prediction. TOGCL leverages a dynamic user-service invocation graph to model historical interactions, providing a comprehensive representation of user-service relationships. Building on this graph, it develops a target-prompt graph attention network to extract online deep latent features of users and services at each time slice, simultaneously considering implicit collaborative relationships between target users/services and their neighbors, as well as relevant historical QoS values. Additionally, a multi-layer Transformer encoder is employed to uncover temporal feature evolution patterns of users and services, leading to temporal-aware QoS prediction. Extensive experiments conducted on the WS-DREAM dataset demonstrate that our proposed TOGCL framework significantly outperforms state-of-the-art methods across multiple metrics, achieving improvements of up to 38.80%. These results underscore the effectiveness of the TOGCL framework for precise temporal QoS prediction.