mlsanityData to 14 Oct 2026

TorchGT: A Holistic System for Large-scale Graph Transformer Training

arXiv 2407.14106 ↗First posted 19 Jul 2024cs.DC cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Graph Transformer is a new architecture that surpasses GNNs in graph learning. While there emerge inspiring algorithm advancements, their practical adoption is still limited, particularly on real-world graphs involving up to millions of nodes. We observe existing graph transformers fail on large-scale graphs mainly due to heavy computation, limited scalability and inferior model quality. Motivated by these observations, we propose TorchGT, the first efficient, scalable, and accurate graph transformer training system. TorchGT optimizes training at different levels. At algorithm level, by harnessing the graph sparsity, TorchGT introduces a Dual-interleaved Attention which is computation-efficient and accuracy-maintained. At runtime level, TorchGT scales training across workers with a communication-light Cluster-aware Graph Parallelism. At kernel level, an Elastic Computation Reformation further optimizes the computation by reducing memory access latency in a dynamic way. Extensive experiments demonstrate that TorchGT boosts training by up to 62.7x and supports graph sequence lengths of up to 1M.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 52 of its 71 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

52 of 71 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  6. 2017Graph Attention Networkscited by 8,923
  7. 2017Inductive Representation Learning on Large Graphscited by 6,320
  8. 2018How Powerful are Graph Neural Networks?cited by 4,393
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  12. 2020Reformer: The Efficient Transformercited by 1,668
38 more
  1. 2020Rethinking Attention with Performerscited by 1,601
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  3. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 881
  4. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 713
  5. 2020Benchmarking Graph Neural Networkscited by 680
  6. 2020A Generalization of Transformer Networks to Graphscited by 661
  7. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 631
  8. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  9. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 584
  10. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 545
  11. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 520
  12. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 506
  13. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 478
  14. 2021Rethinking Graph Transformers with Spectral Attentioncited by 473
  15. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  16. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  17. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 264
  18. 2020The Open Catalyst 2020 (OC20) Dataset and Community Challengescited by 246
  19. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  20. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  21. 2023NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classificationcited by 210
  22. 2022Representing Long-Range Context for Graph Neural Networks with Global Attentioncited by 195
  23. 2022Structure-Aware Transformer for Graph Representation Learningcited by 194
  24. 2020Graph-Bert: Only Attention is Needed for Learning Graph Representationscited by 177
  25. 2023Exphormer: Sparse Transformers for Graphscited by 170
  26. 2019Blockwise Self-Attention for Long Document Understandingcited by 159
  27. 2022NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphscited by 137
  28. 2023SGFormer: Simplifying and Empowering Transformers for Large-Graph Representationscited by 133
  29. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  30. 2021Global Self-Attention as a Replacement for Graph Convolutioncited by 114
  31. 2023DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusioncited by 87
  32. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 83
  33. 2020Computing Graph Neural Networks: A Survey from Algorithms to Acceleratorscited by 81
  34. 2021Gophormer: Ego-Graph Transformer for Node Classificationcited by 63
  35. 2020A Large-Scale Database for Graph Representation Learningcited by 55
  36. 2020GNNAdvisor: An Adaptive and Efficient Runtime System for GNN Acceleration on GPUscited by 51
  37. 2020Interpretable Rumor Detection in Microblogs by Attending to User Interactionscited by 44
  38. 2017Block-Sparse Recurrent Neural Networkscited by 39

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Fused3S: Fast Sparse Attention on Tensor Corescited by 2
  2. 2025Exploring the Global-to-Local Attention Scheme in Graph Transformers: An Empirical Studycited by 0
  3. 2026Automated Tensor-Relational Decomposition for Large-Scale Sparse Tensor Computationcited by 0
  4. 2026Unifying Graph Neural Networks Through a Common Layer Equationcited by 0
  5. 2026MegaGraph: Towards Efficient Training of Large-Scale Graph Transformers with Automated Hybrid Parallelismcited by 0