# LLM-driven Knowledge Distillation for Dynamic Text-Attributed Graphs

arXiv: https://arxiv.org/abs/2502.10914 · HTML: https://mlsanity.com/paper/2502.10914/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 15 Feb 2025.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020)
- [Inductive Representation Learning on Temporal Graphs](https://mlsanity.com/paper/2002.07962/) (2020)
- [Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks](https://mlsanity.com/paper/1908.01207/) (2019)
- [Representation Learning for Dynamic Graphs: A Survey](https://mlsanity.com/paper/1905.11485/) (2019)
- [Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks](https://mlsanity.com/paper/2101.05974/) (2021)
- [Towards Better Dynamic Graph Learning: New Architecture and Unified Library](https://mlsanity.com/paper/2303.13047/) (2023)
- [Do We Really Need Complicated Model Architectures For Temporal Networks?](https://mlsanity.com/paper/2302.11636/) (2023)
- [Foundations and modelling of dynamic networks using Dynamic Graph Neural Networks: A survey](https://mlsanity.com/paper/2005.07496/) (2020)
- [Spatial-Temporal Large Language Model for Traffic Prediction](https://mlsanity.com/paper/2401.10134/) (2024)
- [Temporal Knowledge Graph Completion: A Survey](https://mlsanity.com/paper/2201.08236/) (2022)
- [How Can Large Language Models Understand Spatial-Temporal Data?](https://mlsanity.com/paper/2401.14192/) (2024)
- [DTGB: A Comprehensive Benchmark for Dynamic Text-Attributed Graphs](https://mlsanity.com/paper/2406.12072/) (2024)
- [Distilling Large Language Models for Text-Attributed Graph Learning](https://mlsanity.com/paper/2402.12022/) (2024)
- [UniGLM: Training One Unified Language Model for Text-Attributed Graph Embedding](https://mlsanity.com/paper/2406.12052/) (2024)

## Built on by (most cited first, 8 of 8)

- [Unifying Text Semantics and Graph Structures for Temporal Text-attributed Graphs with Large Language Models](https://mlsanity.com/paper/2503.14411/) (2025)
- [Self-Evolving Agents as Dynamic Graph Transformation: A Survey and New Perspective](https://mlsanity.com/paper/2608.18104/) (2026)
- [Global-Recent Semantic Reasoning on Dynamic Text-Attributed Graphs with Large Language Models](https://mlsanity.com/paper/2509.18742/) (2025)
- [PRISM: Iterative Cross-Modal Posterior Refinement for Dynamic Text-Attributed Graphs](https://mlsanity.com/paper/2605.06073/) (2026)
- [PromptGNN-sim: Deep Fusion and Alignment of GNN and LLMs for Text-Attributed Graph Learning](https://mlsanity.com/paper/2606.30291/) (2026)
- [TTGBench: Benchmarking Topological Evolution and Semantic Drift in Text-attributed Temporal Graphs](https://mlsanity.com/paper/2609.08226/) (2026)
- [TemporalGraphLLM: Temporal Graph Neural Networks with Large Language Models for Dynamic Text-Attributed Graphs](https://mlsanity.com/paper/2609.31881/) (2026)
- [Transfer Learning for Edge Classification on Dynamic Text-Attributed Graphs](https://mlsanity.com/paper/2609.32849/) (2026)

## Abstract (arXiv)

Dynamic Text-Attributed Graphs (DyTAGs) have numerous real-world applications, e.g. social, collaboration, citation, communication, and review networks. In these networks, nodes and edges often contain text descriptions, and the graph structure can evolve over time. Future link prediction, edge classification, relation generation, and other downstream tasks on DyTAGs require powerful representations that encode structural, temporal, and textual information. Although graph neural networks (GNNs) excel at handling structured data, encoding temporal information within dynamic graphs remains a significant challenge. In this work, we propose LLM-driven Knowledge Distillation for Dynamic Text Attributed Graph (LKD4DyTAG) with temporal encoding to address these challenges. We use a simple, yet effective approach to encode temporal information in edges so that graph convolution can simultaneously capture both temporal and structural information in the hidden representations. To leverage LLM's text processing capabilities for learning richer representations on DyTAGs, we distill knowledge from LLM-driven edge representations (based on a neighborhood's text attributes) into saptio-temporal representations using a lightweight GNN model that encodes temporal and structural information. The objective of knowledge distillation enables the GNN to learn representations that more effectively encode the available structural, temporal, and textual information in DyTAG. We conducted extensive experimentation on six real-world DyTAG datasets to verify the effectiveness of our approach LKD4DyTAG for future link prediction and edge classification task. The results show that our approach significantly improves the performance of downstream tasks compared to the baseline models.