# DistTGL: Distributed Memory-Based Temporal Graph Neural Network Training

arXiv: https://arxiv.org/abs/2307.07649 · HTML: https://mlsanity.com/paper/2307.07649/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 14 Jul 2023.

## Builds on (most cited first, 20 of 20)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Temporal Graph Networks for Deep Learning on Dynamic Graphs](https://mlsanity.com/paper/2006.10637/) (2020)
- [Inductive Representation Learning on Temporal Graphs](https://mlsanity.com/paper/2002.07962/) (2020)
- [Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks](https://mlsanity.com/paper/1908.01207/) (2019)
- [EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs](https://mlsanity.com/paper/1902.10191/) (2019)
- [Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks](https://mlsanity.com/paper/2101.05974/) (2021)
- [Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs](https://mlsanity.com/paper/1705.05742/) (2017)
- [DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs](https://mlsanity.com/paper/2010.05337/) (2020)
- [ROLAND: Graph Learning Framework for Dynamic Graphs](https://mlsanity.com/paper/2208.07239/) (2022)
- [Towards Better Evaluation for Dynamic Link Prediction](https://mlsanity.com/paper/2207.10128/) (2022)
- [Variational Graph Recurrent Neural Networks](https://mlsanity.com/paper/1908.09710/) (2019)
- [APAN: Asynchronous Propagation Attention Network for Real-time Temporal Graph Embedding](https://mlsanity.com/paper/2011.11545/) (2020)
- [TGL: A General Framework for Temporal GNN Training on Billion-Scale Graphs](https://mlsanity.com/paper/2203.14883/) (2022)
- [Distributed Hybrid CPU and GPU training for Graph Neural Networks on Billion-Scale Graphs](https://mlsanity.com/paper/2112.15345/) (2021)
- [Instant Graph Neural Networks for Dynamic Graphs](https://mlsanity.com/paper/2206.01379/) (2022)
- [Model-Architecture Co-Design for High Performance Temporal GNN Inference on FPGA](https://mlsanity.com/paper/2203.05095/) (2022)
- [Efficient Scaling of Dynamic Graph Neural Networks](https://mlsanity.com/paper/2109.07893/) (2021)
- [DynamicGEM: A Library for Dynamic Graph Embedding Methods](https://mlsanity.com/paper/1811.10734/) (2018)
- [Efficient Dynamic Graph Representation Learning at Scale](https://mlsanity.com/paper/2112.07768/) (2021)

## Built on by (most cited first, 12 of 12)

- [A Comprehensive Survey of Dynamic Graph Neural Networks: Models, Frameworks, Benchmarks, Experiments and Challenges](https://mlsanity.com/paper/2405.00476/) (2024)
- [A survey of dynamic graph neural networks](https://mlsanity.com/paper/2404.18211/) (2024)
- [Efficient Neural Common Neighbor for Temporal Graph Link Prediction](https://mlsanity.com/paper/2406.07926/) (2024)
- [From Link Prediction to Forecasting: Addressing Challenges in Batch-based Temporal Graph Learning](https://mlsanity.com/paper/2406.04897/) (2024)
- [PGT-I: Scaling Spatiotemporal GNNs with Memory-Efficient Distributed Training](https://mlsanity.com/paper/2507.11683/) (2025)
- [TGM: a Modular and Efficient Library for Machine Learning on Temporal Graphs](https://mlsanity.com/paper/2510.07586/) (2025)
- [Scalable and Efficient Temporal Graph Representation Learning via Forward Recent Sampling](https://mlsanity.com/paper/2402.01964/) (2024)
- [TASER: Temporal Adaptive Sampling for Fast and Accurate Dynamic Graph Representation Learning](https://mlsanity.com/paper/2402.05396/) (2024)
- [Graph Retention Networks for Dynamic Graphs](https://mlsanity.com/paper/2411.11259/) (2024)
- [Towards Ideal Temporal Graph Neural Networks: Evaluations and Conclusions after 10,000 GPU Hours](https://mlsanity.com/paper/2412.20256/) (2024)
- [TAMI: Taming Heterogeneity in Temporal Interactions for Temporal Graph Link Prediction](https://mlsanity.com/paper/2510.23577/) (2025)
- [Mind the truncation gap: challenges of learning on dynamic graphs with recurrent architectures](https://mlsanity.com/paper/2412.21046/) (2024)

## Abstract (arXiv)

Memory-based Temporal Graph Neural Networks are powerful tools in dynamic graph representation learning and have demonstrated superior performance in many real-world applications. However, their node memory favors smaller batch sizes to capture more dependencies in graph events and needs to be maintained synchronously across all trainers. As a result, existing frameworks suffer from accuracy loss when scaling to multiple GPUs. Evenworse, the tremendous overhead to synchronize the node memory make it impractical to be deployed to distributed GPU clusters. In this work, we propose DistTGL -- an efficient and scalable solution to train memory-based TGNNs on distributed GPU clusters. DistTGL has three improvements over existing solutions: an enhanced TGNN model, a novel training algorithm, and an optimized system. In experiments, DistTGL achieves near-linear convergence speedup, outperforming state-of-the-art single-machine method by 14.5% in accuracy and 10.17x in training throughput.